DEV Community: Ethan Linden

Using AI to Write Technical Documentation — What Actually Works

Ethan Linden — Thu, 27 Aug 2026 10:03:57 +0000

An LLM asked to document a payments module — about 900 lines, four public functions, a retry wrapper — will typically produce something gorgeous. Structured headings, a parameters table, a "Common Pitfalls" section. It will also document a timeout_seconds parameter that does not exist. It will describe exponential backoff with jitter when the code does a flat time.sleep(2) in a for loop. It will include a usage example that imports a symbol from the wrong module.

Errors like these survive for weeks, because nobody reads documentation that nobody trusts. The first person to try the example files a bug against the library.

That's the core problem with pointing a model at a repo and asking for docs. The model isn't reading the code so much as pattern-matching it to the average of every similar library it has seen. A retry helper looks like tenacity, so it gets tenacity's parameters. A webhook handler looks like Stripe's, so it gets Stripe's idempotency semantics. The output is fluent, structurally correct, and wrong in exactly the places a reader can't check without reading the source — which is the whole reason they came to the docs.

Wrong docs are worse than no docs. No docs make you read the code. Wrong docs make you skip reading the code.

The reframe: transformation, not generation

Models are bad at inventing facts about a system and good at reformatting facts handed to them. So stop asking for documentation and start asking for transformations of artefacts that already encode the truth.

A codebase is full of these:

Type signatures. def charge(account_id: UUID, amount: Money, *, idempotency_key: str) -> ChargeResult already tells you every parameter, whether it's keyword-only, and what comes back. A model can't invent a fifth parameter if it's constrained to the signature.
Tests. A passing test is an example that ran. Turning test_refund_partial_amount into a documented example is a formatting job, not a reasoning job.
OpenAPI specs / JSON Schema. If the spec is generated from code (FastAPI, tsoa, utoipa), the spec is ground truth. Prose around it is the only thing to write.
Migrations. ALTER TABLE charges ADD COLUMN settled_at timestamptz NULL tells you the column exists, its type, and that it's nullable. That's three facts you don't have to trust anyone for.
Git history. git log --follow -p src/billing/retry.py explains why the sleep is flat. The commit message says "revert jitter, it broke replay determinism in tests." No model would guess that. Every model can summarise it once it's pasted in.

The prompt shifts from "document this module" to "here is the signature, here are the three tests that cover it, here is the commit that introduced it — write the docstring, and write UNKNOWN for anything not present in this input."

That last instruction does most of the work. Models will happily emit UNKNOWN when it's an allowed output. They will not volunteer uncertainty when the only available move is prose.

Work on diffs, not repos

A whole-repo docs pass is a one-time event that produces a large, unreviewable PR. Nobody reads a 4,000-line documentation diff carefully. It gets approved on vibes and rots from day one.

A diff-scoped pass is small enough to actually review, and it lands with the change that motivated it. A script along these lines does the job.

#!/usr/bin/env bash
# scripts/docstring-pass.sh — suggest docstrings for functions touched in this branch
set -euo pipefail

BASE="${1:-origin/main}"
OUT="${OUT:-.docsuggest}"
mkdir -p "$OUT"

# Changed source files, excluding tests and generated code
mapfile -t FILES < <(
  git diff --name-only "$BASE...HEAD" -- '*.py' \
    ':!tests/*' ':!*_pb2.py' ':!**/migrations/*'
)

[[ ${#FILES[@]} -eq 0 ]] && { echo "no source changes"; exit 0; }

for f in "${FILES[@]}"; do
  [[ -f "$f" ]] || continue

  # Tests that mention the module path — cheap but effective
  mod=$(basename "$f" .py)
  tests=$(grep -rl "$mod" tests/ 2>/dev/null | head -3 | xargs -r cat)

  {
    echo "## Current file: $f"
    echo '```

python'; cat "$f"; echo '

```'
    echo "## Diff in this branch"
    echo '```

diff'; git diff "$BASE...HEAD" -- "$f"; echo '

```'
    echo "## Relevant tests (these are the only examples known to run)"
    echo '```

python'; echo "$tests"; echo '

```'
    echo "## Recent history"
    git log --oneline -5 -- "$f"
  } | llm -m claude-sonnet-4-5 \
        --system "$(cat prompts/docstring.md)" \
        > "$OUT/$(echo "$f" | tr '/' '_').md"
done

echo "suggestions in $OUT/"

llm is Simon Willison's CLI (pipx install llm, then llm keys set anthropic). The system prompt is where the constraints live:

<!-- prompts/docstring.md -->
Write Google-style docstrings for functions ADDED or MODIFIED in the diff.
Do not touch functions that are unchanged.

Rules:
- Every parameter you document must appear in the function signature.
  Copy the name and type annotation exactly.
- Do not describe retry, caching, timeout, or concurrency behaviour
  unless it is visible in the code shown.
- Every code example must be derived from a test in the input.
  Cite the test name in a comment. If there is no test, write no example.
- If a parameter's meaning is not determinable from the code, tests,
  or commit messages, write: "UNKNOWN — needs author input".
- Output a unified diff against the current file. No commentary.

Output goes to a scratch directory, not straight into the file. The engineer applies what's right and deletes the rest. The UNKNOWN markers become a to-do list of things only a human knows.

Make examples run

The single highest-value check: every example in the docs must execute in CI. For Python, doctest gets you there almost free.

# .github/workflows/docs.yml
- name: Examples in docstrings must run
  run: python -m pytest --doctest-modules src/ -q

- name: Examples in markdown must run
  run: python -m pytest --codeblocks docs/   # pytest-codeblocks

- name: API docs must match the implementation
  run: |
    python -m app.export_openapi > /tmp/openapi.json
    git diff --exit-code --no-index docs/openapi.json /tmp/openapi.json

That last step is the one that catches drift. If someone adds a query parameter and doesn't regenerate the spec, CI fails with a diff showing exactly what changed. For contract-level checking, schemathesis run docs/openapi.json --url http://localhost:8000 will hammer a running service with requests derived from the spec and report where behaviour and documentation disagree.

The drift trap

Generated prose has no compiler. A docstring that describes behaviour removed six months ago will sit there forever. What's needed is a mechanical link between the doc and the thing it documents.

One approach is to stamp generated sections with a content hash of the source:

<!-- gen-from: src/billing/webhooks.py sha256:4f1a9c2e -->
### Webhook verification
...
<!-- /gen-from -->

And check it:

# scripts/check_docstamps.py
import hashlib, pathlib, re, sys

PAT = re.compile(r"<!-- gen-from: (\S+) sha256:([0-9a-f]+) -->")
stale = []
for md in pathlib.Path("docs").rglob("*.md"):
    for src, want in PAT.findall(md.read_text()):
        got = hashlib.sha256(pathlib.Path(src).read_bytes()).hexdigest()[:8]
        if got != want:
            stale.append(f"{md}: {src} changed ({want} -> {got})")

if stale:
    print("Stale generated docs:\n  " + "\n  ".join(stale))
    sys.exit(1)

It's blunt — a whitespace change trips it — but a noisy check that forces a five-second re-read beats silent rot. Add python scripts/check_docstamps.py to CI and require it on protected branches.

Use the model as a reviewer instead

The highest-value prompt is rarely "write docs." It's this, run against a doc page plus the code it describes:

You are a competent engineer who has never seen this system. Read the documentation, then the code. List every point where the docs would leave you unable to complete the task, and every claim in the docs you cannot verify in the code. Do not rewrite anything.

Typical output from a run against a webhooks page:

"The page says signatures are verified but never says where the secret comes from — env var? config? I'd have to grep."
"The example shows a 200 response. What happens on a duplicate delivery? The code has an on_conflict_do_nothing that the docs don't mention."
"replay_window is documented in seconds. In the code it's compared against a timedelta built from minutes."

The third kind of finding is a real bug, not a documentation nit — surfaced without the model writing a word of documentation. Critique is a much easier task than generation, because the ground truth is in the context window instead of the weights.

Where this still doesn't work

Architecture docs. Anything requiring "why we chose this over that." Runbooks that depend on knowing which alert is a false alarm at 3am. Onboarding guides that need to know what a particular team finds confusing. The model has no access to the arguments in Slack, the outage that shaped the design, or the vendor limitation someone worked around. Generate these and you get a plausible-sounding history that never happened, which is a specific kind of poison for a new hire.

Also: models are bad at knowing what to leave out. Ask for a module overview and you'll get every private helper documented at equal weight to the one function anyone calls.

Whether anyone is actually reading

Producing docs is easy to measure and worthless. These signals are better.

Search queries with zero results. If the docs site has search (Algolia DocSearch, or MkDocs Material's built-in with the plugin's log), the null-result queries are a direct list of pages that should be written. If there are no null-result queries, nobody is searching.

Support questions that have an answer in the docs. Tag them. If a question is answered on a page and someone asked anyway, the page exists but doesn't surface, doesn't rank, or isn't believable. That's three different fixes.

Links in code review. Grep PR comments for the docs domain. Engineers linking each other to a page is the strongest signal that the page is load-bearing.

A canary. Bury one specific, checkable detail in a page — a named constant, an unusual flag. When someone uses it correctly without asking, you know they read it.

Time-to-first-merged-PR for new hires. Slow to move, but it's the number the docs exist to change.

Teams that look honestly at page views over a quarter routinely find a third of their documentation can be deleted without anyone noticing. What deserves to survive is the set of pages updated in the same PR as the code, with examples that run in CI, and a model doing the first pass on the docstrings and a second pass as the confused reader. That combination is worth real engineering time. Pointing the model at the repo and pressing go is not.

ChatGPT vs Claude vs Gemini: How to Actually Choose in 2026

Ethan Linden — Thu, 27 Aug 2026 10:02:57 +0000

The three-way comparison you actually need is not the one on the leaderboards

Pick any two of ChatGPT, Claude and Gemini and there is a benchmark where each one wins. That tells you almost nothing, because the benchmark isn't your codebase, your prompt, your latency budget, or your legal team's stance on data retention.

What does predict the outcome: how each product behaves when it hits the edge of what it knows, whether it can follow the seventh item in a ten-item instruction list, and whether a compliance review will approve the vendor at all. Those are things measurable in an afternoon with prompts already sitting in Jira.

This article is about running that afternoon.

Behavioural differences people consistently report

Treat everything in this section as a hypothesis to test, not a fact. Model behaviour shifts with every release, and the three vendors ship constantly. But these are the patterns developers describe over and over, and each one is checkable.

Long, multi-part instructions. Claude has a reputation for grinding through numbered constraints and often restating them before answering — helpful when you want to see what it thinks the job is, annoying when three lines of code were the goal. ChatGPT tends toward brevity and, under a long constraint list, is more often reported to quietly drop a later item. Gemini sits somewhere in between and sometimes compresses a multi-part request into a summary answer. Test: take a real ticket with eight acceptance criteria and count which criteria appear in the output.

Behaviour when it doesn't know. This is the single most expensive difference. ChatGPT is frequently described as producing a confident, plausible, wrong API call. Claude is more likely to hedge in prose — sometimes so much that the caveats have to be stripped. Gemini with search grounding enabled behaves differently from Gemini without it, which is worth knowing before any comparison. Test: ask all three to use a function that does not exist in your library and see who invents a signature for it.

Refusals and safety. The failure shapes differ operationally, not just tonally. ChatGPT typically returns a short refusal as normal content. Claude tends to explain and offer a partial answer. Gemini can block at the API layer, returning a response with no text and a finish reason indicating safety — which will throw an AttributeError in a parser that assumed response.text always exists. For pipelines processing user-generated content, that difference is a production incident waiting to happen.

Editing files versus writing them. Generating a new module from scratch is the easy case. The hard case is "change these three lines in this 600-line file and leave everything else alone." Some models return the whole file with silent unrelated edits. Some return a diff that doesn't apply. Tools like Aider expose different edit formats (whole-file, unified diff, search/replace) precisely because models differ here. Test with git diff --no-index on the before and after.

Agentic loops and tool use. Each vendor now ships a first-party coding agent: Claude Code, OpenAI's Codex, Gemini CLI. They differ in how many tool calls they'll chain before checking in, how they recover from a failed shell command, and how aggressively they read files before editing. The model and the harness are entangled — Claude inside Cursor is not Claude Code — so evaluate the combination that will actually ship.

Very long inputs. Every vendor advertises a large context window. Advertised capacity and usable capacity are not the same thing. Check the current documented limits directly, because they change, and then check the more important thing: paste an actual repo dump in and ask a question whose answer lives in the middle. That's where degradation shows up.

Characteristic failures. Reported patterns: ChatGPT invents plausible library APIs. Claude adds defensive code and explanatory comments nobody asked for. Gemini wraps JSON in markdown fences after being told not to. All three are fixable with prompting. Which one is cheapest to fix depends on the pipeline.

The harness

Install the three official SDKs:

pip install openai anthropic google-genai
export OPENAI_API_KEY=... ANTHROPIC_API_KEY=... GOOGLE_API_KEY=...
export OPENAI_MODEL=... ANTHROPIC_MODEL=... GEMINI_MODEL=...

Set the model env vars from each vendor's current model list — don't hardcode IDs into a script that'll outlive them.

bench.py:

import os, json, pathlib, random, concurrent.futures as cf
from openai import OpenAI
from anthropic import Anthropic
from google import genai

oai, ant, gem = OpenAI(), Anthropic(), genai.Client()

def chatgpt(system, user):
    r = oai.chat.completions.create(
        model=os.environ["OPENAI_MODEL"],
        messages=[{"role": "system", "content": system},
                  {"role": "user", "content": user}],
    )
    return r.choices[0].message.content, r.usage.model_dump()

def claude(system, user):
    r = ant.messages.create(
        model=os.environ["ANTHROPIC_MODEL"],
        max_tokens=8192,
        system=system,
        messages=[{"role": "user", "content": user}],
    )
    text = "".join(b.text for b in r.content if b.type == "text")
    return text, r.usage.model_dump()

def gemini(system, user):
    r = gem.models.generate_content(
        model=os.environ["GEMINI_MODEL"],
        contents=user,
        config={"system_instruction": system},
    )
    # Gemini can return a blocked candidate with no text at all.
    text = r.text if r.candidates and r.candidates[0].content else ""
    return text, {"finish": str(r.candidates[0].finish_reason)}

RUNNERS = {"chatgpt": chatgpt, "claude": claude, "gemini": gemini}

def main():
    cases = [json.loads(l) for l in open("cases.jsonl")]
    out = pathlib.Path("runs"); out.mkdir(exist_ok=True)
    manifest = {}
    with cf.ThreadPoolExecutor(max_workers=9) as pool:
        futs = {}
        for c in cases:
            for name, fn in RUNNERS.items():
                futs[pool.submit(fn, c.get("system", ""), c["user"])] = (c["id"], name)
        for f in cf.as_completed(futs):
            cid, name = futs[f]
            try:
                text, usage = f.result()
            except Exception as e:
                text, usage = f"<<ERROR {type(e).__name__}: {e}>>", {}
            d = out / cid; d.mkdir(exist_ok=True)
            (d / f"{name}.md").write_text(text)
            manifest.setdefault(cid, {})[name] = usage
    # blind labels so you don't score the logo
    for cid, vendors in manifest.items():
        labels = ["A", "B", "C"]; random.shuffle(labels)
        key = dict(zip(RUNNERS, labels))
        for name, label in key.items():
            (out / cid / f"{name}.md").rename(out / cid / f"{label}.md")
        (out / cid / "key.json").write_text(json.dumps({"key": key, "usage": vendors}))
    print("done ->", out)

if __name__ == "__main__":
    main()

cases.jsonl comes from the backlog, not from a public eval set. Ten to twenty lines like:

{"id": "t-1041", "system": "You are a senior Go engineer.", "user": "Here is handlers/auth.go:\n<paste>\nAdd rate limiting per API key using golang.org/x/time/rate. Keep the existing error envelope. Do not change function signatures. Return only the changed functions."}
{"id": "t-1042", "system": "", "user": "Using our internal client, call billing.ReconcileInvoiceBatch(ctx, ids) and handle partial failure."}

That second case is deliberate — ReconcileInvoiceBatch doesn't exist. It measures hallucination, not correctness.

Score blind, one case at a time, five criteria, 0–2 each:

Instruction adherence — every stated constraint honoured.
Honesty — flags what it can't know instead of inventing it.
Edit fidelity — no unrequested changes; patch applies.
Format compliance — parseable on the first try, no stray fences.
Signal-to-noise — shippable without deleting paragraphs.

Then compare outputs directly:

cd runs/t-1041 && git diff --no-index A.md B.md | head -60

Reveal the key file last. Brand priors are strong, including the scorer's.

For cost, don't guess: the usage dicts captured during the run give real token counts for real prompts. Multiply by each vendor's currently published rates. Do that in a spreadsheet that gets refreshed, because all three change pricing and tiering, and the cheap-tier models — Gemini Flash, the smaller ChatGPT and Claude tiers — often change the answer entirely.

The boring factors that actually decide it

Data retention and training. All three offer API terms that differ from their consumer chat terms, and the consumer ChatGPT, Claude and Gemini apps have their own opt-out settings that are not the same as the API defaults. Zero-retention arrangements exist but usually require asking. Read the current DPA for the specific product and tier being bought — not a blog post, not this one.

Region and deployment. Claude runs on Anthropic's API, AWS Bedrock and Google Vertex AI. ChatGPT models run on OpenAI's API and Azure OpenAI. Gemini runs on Google AI Studio's API and Vertex AI. Where EU-only processing is required, the answer is usually the cloud-hosted variant, and available regions per model are documented per platform. This constraint eliminates options faster than any capability test.

Rate limits. All three tier limits by account maturity and spend. A model that's fast in a notebook can throttle hard on launch day. Log the 429 responses and retry-after headers during the eval to see what's coming, and request increases before launch, not after.

Outages. All three have had them. Check status.openai.com, status.anthropic.com and the Google Cloud status dashboard. Build the fallback on day one: route through LiteLLM or a house adapter so switching vendor is a config change, and pick the fallback from a different vendor — a Gemini fallback for a Claude primary, not a smaller Claude for a bigger one.

Switching cost, honestly. The API call is the easy part. What binds you is everything around it: tool-call schema shapes, structured-output mechanisms, prompt-caching semantics (Anthropic's explicit cache_control blocks versus the other two's approaches), and — most of all — prompts tuned against one model's quirks. The eval suite is the real portability layer. If bench.py can be rerun and re-scored in an hour, switching is annoying. If it can't, switching is a quarter.

Where to start

If the bottleneck is multi-file refactoring in an existing repo, start with Claude via Claude Code, and test ChatGPT through Codex on the same three tickets — the edit-fidelity gap is real and it's measurable in one afternoon.

If the bottleneck is cost per call at volume on short, well-specified tasks, start with Gemini Flash, and test the smaller ChatGPT tier as the fallback — run both through the harness above with the actual token distribution before committing.

If the bottleneck is an existing Azure or GCP footprint with procurement as the long pole, start with whichever of Azure OpenAI or Vertex the org has already signed, and test Claude on Bedrock or Vertex as the second opinion — it can be added without a new vendor contract.

If the bottleneck is hallucinated APIs breaking the pipeline, don't start with a model. Start with the honesty case from cases.jsonl, run all three, and let the scores pick.

AI Is Transforming Software Development Workflows

Ethan Linden — Wed, 26 Aug 2026 20:42:26 +0000

The highest-value use of Claude Code is rarely the one in the demos, and it isn't writing code. It's answering a question like "where does the webhook retry backoff actually get configured?" in a Rails app you've been on for three weeks — and getting it right, pointing at a Sidekiq worker's sidekiq_retry_in block that four different grep attempts missed because the constant was named SPACING.

That's the shape of the real productivity gain. Below is what holds up under daily use across the kinds of codebases these tools get pointed at — a Python service, a Go CLI, a legacy Rails monolith — and the parts that reliably waste time.

Where it genuinely changes the day

Reading code you didn't write. This is the biggest one and it gets undersold because it doesn't produce a diff. Dropping into an unfamiliar module and asking "trace what happens when a POST /v2/orders comes in, list every file involved" gets you a map in under a minute. It's sometimes wrong at the edges, but it's wrong in the way a helpful colleague is wrong — close enough to orient you, and cheap to verify by opening the files it named. Onboarding to a new repo shifts from "read for two days" to "read for two hours with a search assistant."

Mechanical migrations with a clear pattern. Converting 60 test files from unittest to pytest fixtures. Swapping a deprecated moment call for date-fns across a frontend. Adding context.Context as the first parameter to 200 Go functions and threading it through. These are jobs where the transformation is obvious and the tedium is the whole problem. aider is particularly good here because it commits after each file, so git log --oneline gives a reviewable trail and git revert is per-file.

Commit messages and PR descriptions. Small, but it removes a real friction point. aider writes them from the diff by default. Outside an agent session, git diff --cached | claude -p "write a conventional commit message" does the same job. The messages are reliably better than the ones written by hand at 6pm.

The first draft of tests for code that already works. Characterization tests specifically — pinning existing behaviour before a refactor. More on this below, including where it bites.

Debugging as a search problem. Paste a stack trace plus the relevant file and ask "what are three things that could cause this?" The model isn't smarter than the developer; it's just faster at enumerating hypotheses, and it doesn't get attached to the first one.

A worked example: the characterization test loop

A pricing function nobody wants to touch:

# pricing.py
def apply_discounts(cart, customer, now):
    total = sum(i.price * i.qty for i in cart.items)
    if customer.tier == "gold" and total > 100:
        total *= 0.9
    if now.weekday() == 2:  # Wednesday promo, added 2019, never removed
        total -= min(5.0, total * 0.05)
    return round(total, 2)

The loop, with aider:

aider --model sonnet pricing.py tests/test_pricing.py

> Write characterization tests for apply_discounts. Do not change pricing.py.
> Cover: gold tier above and below the 100 threshold, Wednesday vs non-Wednesday,
> and the interaction of both. Use explicit datetime values, no freezegun.
> Assert exact expected floats.

Then, in the same session:

> /run python -m pytest tests/test_pricing.py -q

aider feeds the failures back in and iterates. Two rounds is typically enough to go green — six tests, all passing.

Then the important step, which the tool will not do for you. Green tests prove nothing until they've been proven capable of going red. Break the function on purpose:

-    if now.weekday() == 2:
+    if now.weekday() == 3:

Four of the six tests still pass. Two of them look like this:

def test_gold_customer_gets_discount(mock_pricing_client):
    result = apply_discounts(cart, gold_customer, datetime(2024, 5, 1))
    assert result < 200
    assert mock_pricing_client.called_once

Two problems, both classic. assert result < 200 is a tautology given the inputs. And mock.called_once is not a real attribute — MagicMock happily returns a new child mock, which is truthy, so that line can never fail. The real spelling is assert_called_once(). The called_once mistake is common enough that it turns up in generated tests across unrelated repos.

The fix is to make "can this test fail?" part of the loop, not a thing anyone remembers to do. For Python:

pip install mutmut
mutmut run --paths-to-mutate pricing.py --tests-dir tests/
mutmut results

Any surviving mutant is a test that isn't testing. For a lighter version, git stash the implementation change and confirm red before confirming green. On the Go side, go test -run TestPricing -count=1 plus deliberately inverting a boolean gets 80% of the same signal in ten seconds.

That loop — generate, run, deliberately break, re-run — is the difference between AI-written tests being useful and being coverage theatre.

Where it costs you time

Hallucinated APIs, and specifically plausible ones. Not foo.bar_baz_quux() — that gets caught. It's boto3 calls with a MaxResults kwarg that the particular client doesn't take, or a pandas DataFrame.explode(ignore_index=True) on a version that predates the kwarg, or a Django QuerySet method that exists on the manager but not the queryset. The failure is a TypeError at runtime, in a code path that runs monthly.

Confident-but-wrong refactors. The pattern: you ask for a small change, the agent decides three adjacent things are also wrong, and returns a 400-line diff where the requested change is 12 lines. The other 388 lines look fine. They compile. One of them dropped a nil check. This is the argument for running agents with a hard scope instruction and reviewing every hunk — and why aider's per-change commits matter more than they sound.

Review fatigue. This is the sleeper cost. Reading a diff you didn't write is slower and less reliable than reading one you did, because you don't have the model of why in your head. Generate enough code and you become a full-time reviewer of a colleague who never learns. The common failure is approving an agent's work at the depth normally reserved for a Dependabot bump. That's how the subtle ones get in.

Anything requiring taste. Where the module boundary goes. Whether this should be a queue or a cron. Whether the right fix is to delete the feature. Current models will produce a competent implementation of a bad idea, enthusiastically.

Tool notes, briefly

Copilot's inline completion is still the highest value-per-keystroke thing on the list — it's autocomplete that understands the file, and it costs nothing when it's wrong because you just keep typing. Cursor's agent mode is strong when pointed at exact files and weak when left to search. Claude Code and Codex are better at multi-file work and much better at reading. aider wins on git hygiene. continue.dev with a local model via Ollama is the only option when code can't leave the building; quality is noticeably lower, and it's still worth it for the read-and-explain use case.

What to actually measure

Vendor dashboards will show "suggestion acceptance rate." Ignore it. Accepting a suggestion is not a business outcome; it's a keystroke.

Things that would actually tell you something:

PR size distribution. If median diff size is climbing, review quality is falling, whether or not anyone says so.
Time from PR open to first substantive review comment. Generation is cheap now; review is the bottleneck. If this number is growing, the constraint has moved, not disappeared.
Change failure rate and revert rate, from your deploy tooling. This is the one that catches confident-but-wrong.
Mutation score on new test files, if you can afford to run it in CI weekly. Line coverage will look great and mean nothing.
Time-to-first-merged-PR for new joiners. The onboarding gain is real and this is where it shows up.

And run it as an actual comparison. Pick two similar teams, give one the tooling for a quarter, look at the numbers above. That's not a rigorous trial, but it beats the alternative, which is everyone agreeing it feels faster while the revert rate quietly doubles.