Score Each Doc Section Before a Free Model Drafts It: A Fitness Harness

The useful question about model-drafted documentation is not whether the prose reads well. It is which sections a given model can draft without lowering the floor your team already set. Free model access moves the cost side of that question and leaves the correctness side untouched, so the answer has to come from a repeatable check rather than a read-through. This post builds that check: four deterministic tests per section, a verdict table, and a CI gate that runs on every regeneration.

Why section-level scoring beats file-level review

Doc pages are not uniform units of risk. A glossary entry is enumerable, a how-to is executable, an API reference is schema-bound, and a migration note is judgment-heavy. A model that passes on one of those says almost nothing about the next. Reviewing at the file level hides that difference, because a single weak paragraph forces a full rewrite of a page whose other sections were fine.

Section-level scoring also survives regeneration. Once a model rewrites a page, the interesting question is not "is this text new" but "did anything I own disappear, and did anything unregistered appear". Both are measurable without reading for style. That is the gap a fitness harness fills, and it is deliberately boring work.

Finally, section scores are portable. When you switch models or prompt versions, you keep the same rubric and compare pass rates instead of re-arguing taste in a review thread. The rubric is the contract; the model is a variable.

Step 1: Mark the spans a draft may never alter

Start by declaring ownership boundaries inside the markdown itself, so they travel with the file and survive conversion. Anything between the markers is human-owned text that a regeneration must reproduce byte for byte.

<!-- owner: payments-team -->
Refunds settle in the customer's currency, and partial refunds keep the original fee.
<!-- /owner -->

Two rules keep this marker useful. First, only wrap sentences whose wording was a deliberate decision, not every paragraph, or the check becomes noise. Second, never wrap generated text, because a model that reformats its own output will trip a check it was never meant to fail.

Step 2: Freeze the facts a section may reference

Two small artifacts make the checks possible: an export list of public identifiers and a claims registry keyed by stable IDs. Generate the export list from your build, never by hand, so it always matches the shipped API.

["createClient", "client.retry", "RetryPolicy", "client.timeout"]

{
  "retry-after": {
    "value": "60 s",
    "source": "src/limiter.ts:41",
    "owner": "platform",
    "expires": "2027-01-31"
  }
}

A sentence that states a number, limit, or duration must carry its claim ID inline, for example The header honors a [claim:retry-after] backoff window. If the model writes the number without the tag, the draft fails, and a human adds the tag only after checking the source line.

Step 3: Run four deterministic checks

The harness below is a single dependency-free script. It checks protected spans, unknown identifiers, unregistered numeric claims, and new identifiers introduced by this regeneration.

#!/usr/bin/env python3
"""docfit.py: deterministic fitness checks for a model-drafted markdown section.

Exit code 1 means at least one check failed; CI stops there.
"""
from __future__ import annotations

import argparse
import json
import pathlib
import re
import sys

NUMBER_CLAIM = re.compile(
    r"\b\d+(?:[.,]\d+)?\s*(?:ms|s|min|h|MB|GB|%|x|tokens?|requests?|rps)\b", re.I
)
INLINE_IDENT = re.compile(r"`([A-Za-z_][\w.]{2,}(?:\(\))?)`")
PROTECTED = re.compile(r"<!--\s*owner:[^>]*-->.*?<!--\s*/owner\s*-->", re.S)
CLAIM_TAG = re.compile(r"\[claim:([\w.-]+)\]")
SENTENCE = re.compile(r"(?<=[.!?])\s+")


def read(path: str) -> str:
    return pathlib.Path(path).read_text(encoding="utf-8")


def sentences(text: str):
    for block in re.split(r"\n\s*\n", text):
        for s in SENTENCE.split(block):
            s = s.strip()
            if s:
                yield s


def protected_spans(text: str):
    return [m.group(0) for m in PROTECTED.finditer(text)]


def check_protected(baseline: str, draft: str) -> dict:
    missing = [s for s in protected_spans(baseline) if s not in draft]
    return {"check": "protected_spans", "ok": not missing,
            "detail": [m[:60] for m in missing]}


def check_identifiers(draft: str, exports: set[str]) -> dict:
    seen = {m.group(1).removesuffix("()") for m in INLINE_IDENT.finditer(draft)}
    unknown = sorted(n for n in seen if n not in exports)
    return {"check": "identifiers", "ok": not unknown, "detail": unknown}


def check_claims(draft: str, registry: dict) -> dict:
    problems = []
    for s in sentences(draft):
        if not NUMBER_CLAIM.search(s):
            continue
        tags = CLAIM_TAG.findall(s)
        if not tags:
            problems.append({"why": "unregistered_number", "text": s[:120]})
            continue
        problems += [{"why": "unknown_claim_id", "text": t}
                     for t in tags if t not in registry]
    return {"check": "claims", "ok": not problems, "detail": problems}


def check_drift(baseline: str, draft: str) -> dict:
    """Advisory: identifiers that appear for the first time in this regeneration."""
    old = {m.group(1) for m in INLINE_IDENT.finditer(baseline)}
    new = {m.group(1) for m in INLINE_IDENT.finditer(draft)} - old
    return {"check": "new_identifiers", "ok": True, "detail": sorted(new)}


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("draft")
    ap.add_argument("--baseline", required=True)
    ap.add_argument("--api", required=True)
    ap.add_argument("--registry", required=True)
    ap.add_argument("--out")
    args = ap.parse_args()

    draft, baseline = read(args.draft), read(args.baseline)
    exports = set(json.loads(read(args.api)))
    registry = json.loads(read(args.registry))

    report = [
        check_protected(baseline, draft),
        check_identifiers(draft, exports),
        check_claims(draft, registry),
        check_drift(baseline, draft),
    ]
    failed = [r["check"] for r in report if not r["ok"]]
    payload = {"draft": args.draft, "failed": failed, "report": report}
    if args.out:
        pathlib.Path(args.out).write_text(json.dumps(payload, indent=2), encoding="utf-8")
    print(json.dumps({"draft": args.draft, "failed": failed, "checks": len(report)}))
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

Run it against the previous accepted version, not against an empty file, because three of the four checks are diff-relative.

python tools/docfit.py docs/next/retry.md \
  --baseline docs/main/retry.md \
  --api build/api.json \
  --registry docs/claims.json \
  --out build/docfit-retry.json

Step 4: Convert the report into a per-section verdict

A raw failure list invites negotiation, so map section types to the checks that must pass before a model draft is even eligible for human review. The table below is a starting policy, not a universal standard.

Section type	protected spans	identifiers	claims	Advisory drift	Draft eligibility
Glossary entry	not required	required	required if numeric	review names	Model drafts, human edits
How-to steps	required	required	required	review commands	Model drafts, human signs
API reference	required	required	required	review all	Model drafts, human signs
Migration guidance	required	required	required	review all	Human drafts, model edits
Security or billing caveats	required	required	required	review all	Human only

Encode the policy next to the harness so it is reviewable in the same pull request that changes a doc.

REQUIRED = {
    "glossary": {"identifiers", "claims"},
    "how-to": {"protected_spans", "identifiers", "claims"},
    "api-reference": {"protected_spans", "identifiers", "claims"},
}


def eligible(section_type: str, report: dict) -> bool:
    failed = {r["check"] for r in report["report"] if not r["ok"]}
    return not (REQUIRED.get(section_type, set()) & failed)

A failing section is not a rejected article; it is a section that goes back to the prompt with the failure list attached. That feedback loop is where most of the quality gain comes from, because vague prompt instructions become concrete constraints.

Step 5: Gate every regeneration in CI

Run the harness on each changed markdown file and keep the JSON reports as build artifacts. Reviewers then open the report before opening the rendered page.

name: docs-fitness
on: [pull_request]
jobs:
  docfit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - name: Export public identifiers
        run: node scripts/export-api.js > build/api.json
      - name: Score every regenerated section
        run: |
          for f in $(git diff --name-only origin/main -- 'docs/**/*.md'); do
            git show origin/main:"$f" > build/baseline.md
            python tools/docfit.py "$f" --baseline build/baseline.md \
              --api build/api.json --registry docs/claims.json \
              --out "build/$(basename "$f").docfit.json"
          done
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: docfit-reports, path: build/*.docfit.json }

If a baseline file does not exist, treat the run as a new page and skip the protected-span and drift checks explicitly, rather than letting a missing file fail the job for the wrong reason.

Where free model access and a free server actually fit

Producing several candidate drafts per section is the part of this loop that costs money, and it is also the part where a cheaper model is genuinely sufficient, because the harness filters conformance rather than taste. Free model access is useful here for generating candidates you intend to reject; the scoring, not the generation, is what you keep. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option matters for teams that cannot add a paid runner to a docs repository, since the harness plus a rendered preview can live there instead of on a laptop. Availability is what the operator states it is; the script above has no product dependency and runs anywhere Python does, so treat the hosting as convenience rather than architecture.

Limits, and who should skip this

The harness measures conformance, not truth. It cannot tell you that a section is unnecessary, that an explanation is missing, or that a sentence is technically wrong while still matching the registry. Those remain human responsibilities, and the registry only helps if someone actually checks the source line behind each claim.

Maintenance is the real cost. Every new numeric claim adds a registry entry with an owner and an expiry date, and a stale registry is worse than none because it creates false confidence. Budget roughly one review pass per changed section, and drop the approach entirely if nobody owns that pass.

Skip it when the docs surface is small and stable, when the repository has no CI to attach to, or when documentation is regulated and requires certified review regardless of automated signals. In those cases a protected-span check alone captures most of the value at a fraction of the setup cost.

For a first week, record three numbers per section type: pass rate from the harness, share of numeric sentences carrying a claim ID, and count of protected-span failures. Rising pass rates with a flat registry size usually means the model learned your constraints; rising pass rates with a shrinking registry usually means the checks were quietly relaxed.

If you want to run this loop before committing a model budget or a build box, the free model access and free server option in MonkeyCode are enough to start, and the harness stays in your repository either way.