Reject Compatibility Claims Before Generated Reference Pages Merge

Generated reference pages should describe the current public surface and should not invent promises about future versions. Compatibility ranges, deprecation calendars, and security posture remain human-owned claims that a scanner must reject. A claim gate can enforce that split before any generated markdown file reaches the default branch. The sections below specify a taxonomy, a freeze map, a scanner, and a reproducible test plan.

Why generated pages leak promises

Most drafting prompts ask a model to write helpful documentation from a repository snapshot and a few examples. Helpful language often includes version ranges, migration comfort, and safety statements that the tree does not prove. Those sentences read like product contracts even when the author intended them as casual guidance for readers. A gate that classifies claims is cheaper than reviewing every adjective after the draft already exists.

Recent public threads about agent workflows keep returning to the same failure: the system assumes missing facts instead of stopping. Documentation generation fails in that same way when a model fills silence with reassurance. The gate below treats reassurance as a defect unless a human freeze file already contains the sentence. That rule is mechanical, testable, and independent of any particular model vendor.

Claim taxonomy

Treat every sentence in a generated page as one of three claim classes before the file can merge. SNAPSHOT claims restate symbols, signatures, paths, and tests that exist in the current working tree. CONTRACT claims describe obligations across versions, audiences, or threat models and stay under human authorship. SILENCE marks topics the page must not discuss, including untested error paths and unpublished roadmap items.

Class	Model may draft	Human must own	Typical leak phrases
SNAPSHOT	Yes, with a path citation	Citation list and heading set	function, parameter, returns, defined in
CONTRACT	No	Compatibility, deprecation, support	backward compatible, will remain, supported on
SILENCE	No	Absence itself	always safe, cannot fail, coming soon

The table is a review artifact, not a linguistic theory, and reviewers should extend the phrase list per repository. Legal product names, license interpretations, and incident-response language belong in CONTRACT or SILENCE, never in SNAPSHOT. If a sentence cannot be checked against a path in the freeze map, the gate classifies it as CONTRACT by default. Defaulting to CONTRACT is conservative and keeps false SNAPSHOT labels from shipping.

Freeze map format

Store the classification outside the model prompt so later drafts cannot renegotiate ownership of a heading. The freeze map lists each intended heading, its class, and the files that may be cited as evidence. CONTRACT and SILENCE headings are either omitted from generation or filled from a human-owned stub file. SNAPSHOT headings may be drafted, but only with citations that resolve to paths recorded in the map.

Proposed freeze map, checked in beside the docs tree and reviewed like code:

# docs/freeze-map.yaml  (proposed local artifact, not a live service)
version: 1
pages:
  - path: docs/reference/queue.md
    headings:
      - id: enqueue-signature
        class: SNAPSHOT
        cite:
          - src/queue.py
          - tests/test_queue.py
      - id: retry-behavior
        class: CONTRACT
        stub: docs/stubs/retry-behavior.md
      - id: threat-model
        class: SILENCE

A heading with class SILENCE must not appear in generated markdown, even as a placeholder section titled later. CONTRACT headings may appear only when the stub file already exists and the generator copies it verbatim. SNAPSHOT headings may be rewritten on each run, provided every citation still resolves. Changing a class is a human review event, not a prompt tweak.

Step 1: Inventory the public surface

Build a read-only inventory from the same files the freeze map is allowed to cite. The inventory is a list of symbols and test names, not a narrative, and it should be generated by ordinary repository tools. Proposed command sequence for a Python library layout:

# Proposed local commands. Review paths before running in CI.
rg -n "^def |^class " src tests > /tmp/surface-inventory.txt
python3 tools/claim_gate.py inventory \
  --freeze docs/freeze-map.yaml \
  --inventory /tmp/surface-inventory.txt \
  --out /tmp/surface.json

The inventory script should fail when a freeze citation points at a missing path. It should also fail when a SNAPSHOT heading cites zero files, because that heading has no evidence boundary. Do not feed the model a glob of the entire repository if the freeze map names a smaller cite set. Narrow inputs reduce the chance that a nearby README becomes an accidental compatibility source.

Step 2: Mark each heading before any draft

Walk the freeze map in review and refuse generation until every heading has exactly one class. Numbered checklist for the human owner of the page:

1. Confirm the page path matches a file the docs pipeline is allowed to write.
2. Assign SNAPSHOT only when a reader can verify the sentence against a cited path.
3. Assign CONTRACT when the sentence would still matter after the cited file changes.
4. Assign SILENCE when the repository lacks tests, schemas, or incident notes for the topic.
5. Record the reviewer identity in the pull request, not inside the model prompt.

Headings that describe current signatures are usually SNAPSHOT. Headings that tell integrators what will not break across releases are CONTRACT. Headings that discuss abuse cases without a checked threat model are SILENCE until that model exists. Mixing those three in one generated paragraph is the defect the later scanner is built to catch.

Step 3: Constrain the draft to SNAPSHOT headings

Pass the inventory JSON and the SNAPSHOT heading list into the drafting step, and omit CONTRACT stubs from the prompt. Instruct the model to restate signatures and test names, and to stop when a requested heading is missing from the SNAPSHOT list. Label the following prompt block as a proposed template, not as a measured production configuration.

Proposed drafting constraints:
- Draft only headings listed with class SNAPSHOT.
- Each sentence must end with a cite token: (cite: path[:line]).
- Do not write version ranges, deprecation dates, or security posture.
- If a symbol is absent from the inventory, write STOP and the symbol name.
- Do not copy CONTRACT stub files; the pipeline inserts those after the scan.

If you already run drafting on a shared workstation, MonkeyCode's free model access and free server option can host that constrained draft plus the gate without adding a paid inference path. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The gate itself is ordinary Python and does not depend on that host remaining available.

Step 4: Scan the draft for leaked claims

After the model returns markdown, scan the file before concatenating human stubs. The scanner has two jobs: reject DENY phrases, and reject citations that are not in the freeze map. Proposed module tools/claim_gate.py:

# Proposed local scanner. Unexecuted example; extend phrases per repo.
from __future__ import annotations

import re
from pathlib import Path

DENY = re.compile(
    r"\b(backward compatible|forwards compatible|will remain|"
    r"supported on|will be removed|not vulnerable|always safe|"
    r"cannot fail|coming soon|sla|guarantee[sd]?)\b",
    re.I,
)
CITE = re.compile(r"\(cite:\s*([^)]+)\)")

def scan_snapshot(markdown: str, allowed_paths: set[str]) -> list[str]:
    errors: list[str] = []
    for i, line in enumerate(markdown.splitlines(), 1):
        if DENY.search(line):
            errors.append(f"L{i}: CONTRACT/SILENCE language in SNAPSHOT draft")
        for raw in CITE.findall(line):
            path = raw.split(":", 1)[0].strip()
            if path not in allowed_paths:
                errors.append(f"L{i}: citation not in freeze map: {path}")
        if line.strip() and not CITE.search(line) and not line.startswith("#"):
            errors.append(f"L{i}: SNAPSHOT sentence missing cite token")
    return errors

def allowed_paths_for(page: dict) -> set[str]:
    paths: set[str] = set()
    for heading in page["headings"]:
        if heading["class"] == "SNAPSHOT":
            paths.update(heading.get("cite") or [])
    return paths

Every non-heading SNAPSHOT sentence needs a cite token so reviewers can jump to evidence without rereading the prompt. CONTRACT language inside a SNAPSHOT draft is a hard failure even when the rest of the page looks accurate. Missing citations are also hard failures, because uncited sentences cannot be distinguished from invented comfort language. The scanner should print line numbers; it should not try to rewrite the draft in place.

Step 5: Fail CI, then insert human stubs

Run the scanner in continuous integration and only then concatenate CONTRACT stubs. Proposed pipeline fragment:

python3 tools/claim_gate.py scan \
  --freeze docs/freeze-map.yaml \
  --draft /tmp/queue.generated.md \
  --page docs/reference/queue.md
# Non-zero exit stops the job before stubs are copied.
python3 tools/claim_gate.py stitch \
  --freeze docs/freeze-map.yaml \
  --draft /tmp/queue.generated.md \
  --out docs/reference/queue.md

Stitching after a clean scan keeps human-owned paragraphs from being re-tokenized by the model on the next run. The generated SNAPSHOT body can churn when signatures change; the stub files should change only in review. If stitch runs before scan, a leaked compatibility sentence can hide beside a correct stub and survive casual diffs. Order is part of the control, not a cosmetic makefile detail.

Reproducible test plan

The tests below are proposed fixtures for the scanner, not measurements from a production docs corpus. Each case should fail closed.

# tests/test_claim_gate.py  (proposed)
from tools.claim_gate import scan_snapshot

ALLOWED = {"src/queue.py", "tests/test_queue.py"}

def test_signature_line_passes():
    md = "enqueue(item) appends to the tail. (cite: src/queue.py:12)\n"
    assert scan_snapshot(md, ALLOWED) == []

def test_compat_phrase_fails():
    md = "enqueue is backward compatible. (cite: src/queue.py:12)\n"
    errors = scan_snapshot(md, ALLOWED)
    assert any("CONTRACT" in e for e in errors)

def test_unknown_citation_fails():
    md = "enqueue retries forever. (cite: README.md)\n"
    errors = scan_snapshot(md, ALLOWED)
    assert any("not in freeze map" in e for e in errors)

def test_missing_cite_fails():
    md = "enqueue appends to the tail.\n"
    errors = scan_snapshot(md, ALLOWED)
    assert any("missing cite token" in e for e in errors)

Add a fixture that includes a CONTRACT stub containing the phrase backward compatible and assert the stitch step copies it unchanged. That case documents the intended split: the same phrase is illegal in generated SNAPSHOT lines and legal in reviewed stubs. Without that positive case, a future refactor might ban the phrase globally and break human-owned compatibility notes. Keep the fixture small enough that reviewers can read it in one screen.

Limitations

Phrase lists are incomplete by construction and will miss novel wording for the same forbidden claim. Models can restate a compatibility promise without using any token in the DENY list, especially with hedged verbs. The cite-token rule does not prove that the cited line supports the sentence; it only proves the path was pre-approved. Humans still read SNAPSHOT drafts for inverted conditions, dropped parameters, and tests that no longer match the source.

The freeze map can rot when files move and citations are not updated in the same change. Inventory generation from regular expressions will miss macros, generated clients, and language features the pattern author did not encode. This workflow also assumes a single default branch narrative and does not model multiple supported major versions as SNAPSHOT data. Multi-version support matrices remain CONTRACT documents even when every version tag exists in the repository.

Who should not use this approach

Teams without a named freeze-map reviewer should not generate reference pages this way, because the map becomes another unowned prompt. Public SDKs that already publish contractual compatibility should not let a SNAPSHOT draft anywhere near those pages. Security-sensitive APIs should keep threat models in SILENCE until a human process exists, rather than hoping a deny list catches every safety adjective. Marketing sites, changelog blogs, and tutorial narratives need different controls and will look stilted under cite tokens.

If the documentation set is smaller than the freeze-map overhead, write the pages by hand and skip generation entirely. The claim gate is a brake for teams that already generate reference text, not a reason to start generating it. When in doubt, leave the heading in SILENCE and ship no paragraph. Empty sections are cheaper to correct than compatibility sentences that readers quote later.

The practical close is to merge the freeze map and the scanner before enabling any drafting job on a shared server. Once those two artifacts exist, a free model path is optional compute, not the source of ownership. Keep compatibility language in stub files that models never open.