DEV Community

Avery Lin
Avery Lin

Posted on

Assign a Drafter Role to Every Doc Path Before Models Generate Walkthroughs

Generated API documentation fails when one writer both invents examples and asserts invariants that operations cannot prove. Path-level ownership is a more reliable control than a longer system prompt that begs the model to be careful. Models can draft walkthroughs, sample payloads, and schema narration that reviewers can check against a committed specification file. Humans still need to own deprecation calendars, authorization side effects, and any claim that would still be wrong if the examples were perfect.

Why a single drafter role collapses

Mixed drafts look coherent in review because prose, tables, and curl samples share the same voice after one generation pass. The failure mode is not grammar; it is a human-owned promise sitting inside a file that the next model run is allowed to rewrite. A walkthrough that misnames a field is cheap to catch against OpenAPI, because the schema is already in git. A paragraph that dates a deprecation, grants a scope, or says an error never occurs is expensive, because no generator output can stand in for product intent.

Teams that skip a drafter map usually compensate with chat instructions such as “do not invent SLAs” or “keep the security section unchanged.” Those instructions do not survive a recursive edit of docs/, a regenerated README, or a merge that treats the model branch as documentation truth. The practical control is mechanical: list paths, assign a role, and reject writes that cross the role boundary.

What a model may draft versus what a human must own

Use the table as a review contract, not as a style preference. If a row cannot be falsified from a committed artifact, it does not belong in a model-writable path.

Doc artifact Model may draft? Human must own Falsify against
Request and response walkthroughs Yes, from examples and schemas No, except product names OpenAPI examples and schemas
curl or HTTP samples Yes, when the spec already has examples Transport and auth header policy Spec examples plus server stubs
Field narration Yes, restating schema types and enums Semantic aliases that the spec omits properties and required
Deprecation calendar No Sunset dates and replacement routes Release board, not the model
Authorization side effects No Scopes that expose or mutate data IdP config and policy tests
Totality claims (“never”, “always”) No Error and empty-state policy On-call docs and incident history
Install one-liners Yes, copied from package manifests Supported runtime matrix lockfiles and CI images
Breaking-change rationale Outline only, labeled proposal Customer-facing why and when Versioning RFC

The middle column is the entire method. Models are useful where a reviewer can replay the text against files that already passed review. Humans remain the owner wherever a correct sample would still leave a false guarantee in the paragraph.

Encode ownership as a path manifest

Keep the contract next to the docs tree so generation jobs and CI read the same file. The manifest below is a proposal for a public HTTP API repository; labels are roles, not model product names.

# docs/ownership.yaml — proposal, not an executed production config
version: 1
roles:
  model_drafter:
    may_write:
      - docs/walkthroughs/**/*.md
      - docs/generated/examples/**/*.http
      - docs/generated/fields/*.md
    must_not_assert:
      - deprecation_dates
      - authz_side_effects
      - totality_claims
  human_owner:
    exclusive_write:
      - docs/invariants/deprecations.md
      - docs/invariants/authz.md
      - docs/invariants/error-policy.md
      - docs/support/runtime-matrix.md
    review_required: true
generation:
  allowed_sources:
    - openapi.yaml
    - package.json
    - src/http/handlers/
  forbidden_sources:
    - chat_transcripts/
    - unverified_metrics.md
Enter fullscreen mode Exit fullscreen mode

Treat may_write as a rewrite permit for a single branch, not as a license to invent missing fields. Treat exclusive_write as a merge blocker when the changing commit is labeled as model output. If a walkthrough needs a date or a scope name, the draft should link to the human-owned file instead of inlining the promise.

Workflow: six numbered gates

  1. Inventory every documentation path that a release ships, including README fragments that duplicate the API surface. Record the source of truth for each path in one spreadsheet column before any model run starts.
  2. Classify each path with the table above, requiring a falsification artifact for every model-writable file. If the artifact is missing, move the path to human_owner until the spec or tests exist.
  3. Commit docs/ownership.yaml on the default branch and refuse generation jobs that cannot load it. A missing manifest should fail closed rather than defaulting to “the model may edit docs/”.
  4. Run the drafter only against may_write globs, with the specification and package manifests as the only input files. Label any rationale, migration story, or “why we broke this” section as a proposal in the output.
  5. Open a pull request that cannot include diffs under exclusive_write. Reviewers then check walkthroughs against openapi.yaml instead of debating tone or confidence.
  6. Merge human-owned edits on a separate pull request, with the same reviewers who own the invariant. Do not squash a model walkthrough and a deprecation date into one commit that CI cannot classify.

The order matters because classification without a gate is still a prompt. Once the permit file is in git, generation becomes a bounded rewrite, and review effort moves to the paths that actually carry product risk.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The generation step in gate four can run on MonkeyCode's free model access and free server option when you want a scratch drafter that never receives the exclusive-write tree. Keep quotas, hardware, and duration out of the docs; those values are not specified here and must not be drafted as if they were measured.

A gate that CI can run

The checker reads the manifest, inspects a base-to-head diff, and fails when a model-labeled commit touches exclusive paths. It is a path filter, not a language model, and it should stay that way.

#!/usr/bin/env python3
"""Fail a model-labeled diff that writes human-owned documentation paths."""
from __future__ import annotations

import os
import subprocess
import sys
from pathlib import Path

import yaml  # PyYAML; install in CI, not from an unverified chat snippet


def git_changed_files(base: str, head: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...{head}"],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def load_exclusive(manifest: Path) -> list[str]:
    data = yaml.safe_load(manifest.read_text())
    return list(data["roles"]["human_owner"]["exclusive_write"])


def matches(path: str, pattern: str) -> bool:
    # fnmatch is enough for ** if you expand it; keep patterns explicit in v1.
    from fnmatch import fnmatch

    return fnmatch(path, pattern) or path == pattern


def main() -> int:
    if os.environ.get("DOC_DRAFTER_ROLE") != "model_drafter":
        return 0  # human commits are reviewed by people, not this gate
    base = os.environ["DOC_DIFF_BASE"]
    head = os.environ["DOC_DIFF_HEAD"]
    exclusive = load_exclusive(Path("docs/ownership.yaml"))
    violations = []
    for path in git_changed_files(base, head):
        for pattern in exclusive:
            if matches(path, pattern):
                violations.append(path)
    if violations:
        print("model_drafter touched human-owned paths:")
        for path in violations:
            print(f"  - {path}")
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())
Enter fullscreen mode Exit fullscreen mode

Wire it so the role comes from the job definition, not from the model’s self-description. A typical CI fragment looks like the following proposal; adjust image names to whatever your org already pins.

# .github/workflows/doc-ownership.yml — proposal
name: doc-ownership
on:
  pull_request:
jobs:
  path-gate:
    runs-on: ubuntu-latest
    env:
      DOC_DRAFTER_ROLE: ${{ contains(github.event.pull_request.labels.*.name, 'model-drafter') && 'model_drafter' || 'human' }}
      DOC_DIFF_BASE: ${{ github.event.pull_request.base.sha }}
      DOC_DIFF_HEAD: ${{ github.event.pull_request.head.sha }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: pip install pyyaml
      - run: python tools/check_doc_ownership.py
Enter fullscreen mode Exit fullscreen mode

Add a second, cheaper check on model-writable files: every curl example must use a path that exists under paths in openapi.yaml. That test catches invented routes without pretending the model understood authorization. Keep totality language out of those files entirely; a linter that greps never, always, and guaranteed on docs/walkthroughs is enough for a first version.

# Fail walkthroughs that smuggle invariant language into model-writable prose.
rg -n -i -e '\bnever\b' -e '\balways\b' -e '\bguaranteed\b' docs/walkthroughs \
  && echo "invariant language in model-writable docs" && exit 1 \
  || true
Enter fullscreen mode Exit fullscreen mode

Limitations

The manifest does not make generated samples correct; it only stops samples from overwriting invariants. A model can still emit a walkthrough that matches the spec and disagrees with production if the spec is stale. Path filters also miss inlined promises, such as a deprecation date pasted into an otherwise allowed walkthrough, unless you add the language linter or a link-only rule.

This workflow assumes a reviewed OpenAPI file, or an equivalent schema, is already the source of truth. It does not replace legal review, security advisory writing, or status-page language during an incident. It also does not measure generator quality, latency, or cost, and no benchmark from this article should be copied into product docs.

Who should not use this approach

Skip the path gate if your documentation is a single narrative README with no schema, because there is no falsification artifact for the model-writable half. Skip it if the model is expected to draft security advisories, pricing, or support windows; those remain human-owned even when the prose looks like a walkthrough. Skip it if your release process cannot label pull requests by drafter role, since unlabeled model commits will either bypass the gate or block legitimate human edits.

Teams that already keep examples in the spec and invariants in a short, hand-edited set of files will get the most value. The useful output is a smaller review surface, not a fully automatic documentation site. If you apply the manifest on a public API repo, keep exclusive-write paths off the model branch and review those files as product changes rather than as generated text.

Top comments (0)