DEV Community

Avery Lin
Avery Lin

Posted on

Section-Level Draft Permissions: A CI Gate for Model-Drafted Documentation

The fastest way to keep model-drafted documentation honest is not to ban the model at all, but to declare which kinds of claims it may write inside each section type and fail the build when it writes anything else.

The squeeze: generation got cheap, verification did not

A recurring theme in developer writing at the moment is that models produce prose far faster than teams can check it, and that tooling keeps improving while the surrounding process does not. Documentation is where that gap surfaces first, because one wrong sentence in a README turns into a support ticket within a day. A reviewer facing a four-hundred-line generated page has two bad options: skim and approve, or rewrite everything by hand. Section-level draft permissions replace that choice with a narrow, testable question about individual claims.

Stop reviewing pages; review claims

Every documentation page mixes section types with wildly different risk levels. A conceptual overview can be vague and still useful, while a numeric limit or a security statement has to be exactly right. Reviewing the whole page treats both the same way, which burns attention on harmless paragraphs and under-protects the dangerous ones. The fix is to attach permissions to sections and to claim types, then let a script enforce those permissions on every pull request.

The claim-scope matrix

Section type Model may draft Human must own Gate enforcement
Overview / concept Framing prose, inside markers Scope claims about who the feature is for Unregistered absolutes fail
Quickstart narration Narration around a repository example The claim that the example is complete Absolutes fail
Numeric limits and latency Nothing Every number and its unit Metric allowlist
Version compatibility Prose, but not version strings Which versions are actually supported Version allowlist
Error catalogue Descriptions of known failures Status codes and trigger conditions Status allowlist
Security / data handling Nothing The entire section Any draft marker there fails review

The matrix is the human-owned artifact in this workflow, and the gate only proves that a claim was registered by somebody. Registration is not truth, but it turns an invisible judgment call into a visible one, which is what a reviewer can realistically audit at scale.

Step 1 - Register the facts a person owns

Start with one file that lists every claim a human is willing to stand behind, written in the exact surface form you expect to see in the docs. Keep it deliberately small, and review edits to it the way you review a database migration, because it is the file that grants the model permission to speak.

{
  "metrics": ["240 ms", "12 %"],
  "versions": ["v2.4.1"],
  "statuses": ["404"],
  "approved_phrases": ["returns 404 when the key is unknown"]
}
Enter fullscreen mode Exit fullscreen mode

Surface-form matching matters more than it looks. If your prose says 240ms and the allowlist says 240 ms, the gate will report a failure even though the number is correct, so pick one spelling and enforce it in the fact file.

Step 2 - Mark the writable region inside each section

Wrap only the paragraph the model may write, using HTML comments that survive Markdown rendering and disappear from the published page.

## Rate limits

<!-- draft:model -->
The API applies per-key limits, and typical reads sit near 240 ms at p95.
Version v2.4.1 introduced the retry header used by the examples below.
<!-- /draft:model -->

See the error catalogue for the response shape used on rejection.
Enter fullscreen mode Exit fullscreen mode

Three rules keep this honest. Prose outside the markers counts as human-authored and is ignored by the gate, a security section must contain zero markers, and a marker pair that wraps a whole page is a signal that the page was never really reviewed by a person.

Step 3 - Enforce the boundary with a small script

The gate below has no dependencies and runs on plain Node, which means it can live in the repository next to the docs instead of in someone's head.

#!/usr/bin/env node
// gate.mjs - fails when a model-drafted block asserts an unregistered claim.
// Usage: node gate.mjs docs/ extra-file.md
import { readFileSync, readdirSync, statSync } from 'node:fs';
import { join, extname } from 'node:path';

const facts = JSON.parse(readFileSync('facts.json', 'utf8'));
const LISTS = {
  metric: new Set(facts.metrics ?? []),
  version: new Set(facts.versions ?? []),
  status: new Set(facts.statuses ?? []),
  absolute: new Set((facts.approved_phrases ?? []).map((s) => s.toLowerCase())),
};

const PATTERNS = [
  ['metric', /\b\d+(?:\.\d+)?\s?(?:ms|s|MB|GB|%|rps|tokens)\b/g],
  ['version', /\bv?\d+\.\d+\.\d+(?:-[0-9A-Za-z.]+)?\b/g],
  ['status', /\b[45]\d\d\b/g],
  ['absolute', /\b(guarantees?|always|never|unlimited|supports all)\b/gi],
];

function markdownFiles(inputs) {
  const out = [];
  for (const input of inputs) {
    if (statSync(input).isDirectory()) {
      for (const name of readdirSync(input)) {
        if (extname(name) === '.md') out.push(join(input, name));
      }
    } else if (extname(input) === '.md') {
      out.push(input);
    }
  }
  return out;
}

function draftedBlocks(markdown) {
  const blocks = [];
  const re = /<!--\s*draft:model\s*-->([\s\S]*?)<!--\s*\/draft:model\s*-->/g;
  let match;
  while ((match = re.exec(markdown)) !== null) {
    const line = markdown.slice(0, match.index).split('\n').length;
    blocks.push({ text: match[1], line });
  }
  return blocks;
}

let failures = 0;
for (const file of markdownFiles(process.argv.slice(2))) {
  const blocks = draftedBlocks(readFileSync(file, 'utf8'));
  for (const block of blocks) {
    for (const [kind, pattern] of PATTERNS) {
      for (const hit of block.text.matchAll(pattern)) {
        const claim = hit[0].trim();
        const key = kind === 'absolute' ? claim.toLowerCase() : claim;
        if (!LISTS[kind].has(key)) {
          failures += 1;
          console.error(`${file}:${block.line} unregistered ${kind}: "${claim}"`);
        }
      }
    }
  }
}

console.log(failures === 0 ? 'gate: ok' : `gate: ${failures} unregistered claim(s)`);
process.exit(failures === 0 ? 0 : 1);
Enter fullscreen mode Exit fullscreen mode

The script does one thing: it finds marked blocks, extracts candidate claims, and checks each one against the allowlist that a person signed off on.

Step 4 - Prove the gate fails before you trust it

A gate that has never failed is indistinguishable from a gate that does nothing, so build three fixtures and record the expected output as a test plan you can re-run after every change.

mkdir -p docs/fixtures

printf '%s\n' '<!-- draft:model -->' \
  'We handle 5000 rps and always return 200.' \
  '<!-- /draft:model -->' > docs/fixtures/bad.md

printf '%s\n' '<!-- draft:model -->' \
  'Typical reads sit near 240 ms on v2.4.1.' \
  '<!-- /draft:model -->' > docs/fixtures/good.md

node gate.mjs docs/fixtures
Enter fullscreen mode Exit fullscreen mode
$ node gate.mjs docs/fixtures
$ echo $?
Enter fullscreen mode Exit fullscreen mode

The failing run should print three findings for bad.md - an unregistered metric, an unregistered status code, and an unregistered absolute - and exit with status 1. Remove bad.md and the same command should print gate: ok and exit with status 0. Run this pair in CI with a run: node gate.mjs docs step, and add facts.json itself to a CODEOWNERS rule so a claim cannot be registered without a named human approving the diff.

Step 5 - Keep the evidence where reviewers already look

Post the gate summary as a pull-request comment next to the prose diff, because a reviewer who has to open a second tool to see which claims were unregistered will simply approve the page instead. If you need a shared preview environment, a small always-on box is enough: the gate is a single Node process and the preview is a static build. Hosting both on one free server option keeps the failure output in the same place the reviewer reads the rendered page, which shortens the loop from minutes of context switching to a few seconds of reading.

Where a free model allowance fits, and where it does not

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The drafting step in Step 2 is the only place where a hosted model genuinely participates, and it is also the least valuable work in this workflow: turning a registered fact list into readable paragraphs. That is exactly the kind of task worth spending free capacity on. MonkeyCode's operator describes a free model allowance, currently stated as 10 million tokens, together with a free server option that can host the preview build and the gate runner, so a team can test this workflow end to end without buying infrastructure first. Treat those terms as time-sensitive, because allowances and capacity on any free tier change faster than documentation does, and check the current page before you plan around them. If you want to try the drafting half yourself, the project is open source on GitHub and you can start with a small facts file of your own rather than a full page migration.

Limitations and who should not adopt this

The gate verifies registration, never correctness, so a wrong number that a human registered will pass every run and still mislead readers. Regex extraction is brittle for hedged prose such as "a couple of hundred milliseconds", and it cannot see claims expressed as diagrams, tables, or screenshots. The workflow also assumes a stable release process, because facts.json needs a source of truth to copy from. Skip it if your documentation is a legal artifact where every sentence is reviewed by counsel, if a single writer owns a small docs set that fits in one head, or if you have no CI at all, since the matrix then becomes another document nobody maintains. For everyone else it is a cheap way to let a model write the easy 70 percent while a person keeps the claims that would actually hurt to get wrong.

Top comments (0)