Refuse Incomplete Evidence Packets Before Generating Reference Docs

Generated reference documentation stays honest only when the model never sees more evidence than a sealed packet contains. Hand the model a repository and it will fill every heading, including headings that have no schema, fixture, or observed status code. Hand it a completeness-scored packet and it can render field tables, examples, and status lists without inventing obligations. Anything the packet cannot support becomes a human-owned gap rather than a fluent paragraph that later fails review.

Why repository-wide prompts fail a verifier

Whole-repository prompts mix three jobs that do not share a verifier: shape inventory, intended behavior, and external promises. Shape inventory can be checked against OpenAPI documents, JSON Schema files, and hashed fixtures in tests. Intended behavior and external promises require a named owner because no file hash can confirm a support window. When those jobs share one prompt, the completion looks complete while unverifiable sentences hide in ordinary-looking Markdown.

A second failure mode is stale narration after a small schema change lands in the default branch. The model still remembers yesterday's field list from a previous chat and writes that list with confidence. A packet that embeds source_sha and fixture_hash makes that memory irrelevant for the renderer. The renderer may print only keys present in the current packet, and PARTIAL packets stop generation instead of blending contexts.

Packet fields versus human-owned claims

Build each packet around a single operation rather than around a chapter of the product manual. The packet may include operation_id, schema pointers, a fixture path, a fixture hash, observed status codes, and the producing git SHA. The packet must not include roadmap copy, severity opinions, customer promises, or calendars that describe future compatibility. Those claims are not extractable from a schema, so they are not model-draftable under this workflow.

Use three completeness classes and keep the assignment mechanical rather than an editorial judgment of prose. COMPLETE means every required field is present and every referenced path hashes against the working tree. PARTIAL means the operation exists but a schema, fixture, or observed status list is still missing. EMPTY means the collector could not verify paths, and no model call should be scheduled.

Decision table for draft permission

Packet class	Required fields present	Model may produce	Human must own
COMPLETE	operation_id, schemas, fixture_hash, status codes, source_sha	Field tables, hashed examples, status lists	SLA, support, compatibility
PARTIAL	operation_id with an incomplete subset	A gap list only	Missing evidence files
EMPTY	Unverifiable or absent paths	Nothing	Packet construction

The table is the policy. If a page needs a sentence that the table routes to a human, that sentence does not belong in the generated file.

Five-step workflow

The sequence below is a documentation compile, not a chat transcript that happens to emit Markdown. Each step has a file output so a reviewer can see why generation ran or why it refused. Skip a step and the later linter will either fail closed or, worse, accept invented claims. Run the steps in order in CI, and keep local chat experiments off the default branch.

1. Collect machine-readable inputs only

Keep a sidecar JSON file per operation rather than pouring a full specification into the model prompt. The collector reads local files, hashes the fixture body, and writes a packet beside a gap file. Treat the following script as a worked example that runs with the Python standard library. Do not pass README text into the packet even when that README is currently accurate.

{
  "operation_id": "createCharge",
  "source_sha": "9f3c1aa",
  "request_schema": {
    "properties": {
      "amount_cents": {"type": "integer", "description": "Integer cents to charge."},
      "currency": {"type": "string", "description": "ISO 4217 currency code."}
    }
  },
  "response_schema": {
    "properties": {
      "id": {"type": "string", "description": "Server-assigned charge identifier."},
      "status": {"type": "string", "description": "Lifecycle value observed in fixtures."}
    }
  },
  "fixture_path": "tests/fixtures/create_charge.json",
  "fixture_hash": "sha256:replace-with-real-hash",
  "observed_status_codes": [201, 400, 409]
}

README sentences often mix inventory with promises, and the model cannot separate them once they share a string. If a human needs README context, that human should close gaps by adding schemas or fixtures, not by widening the prompt. Widening the prompt reintroduces the verifier problem that the sealed packet exists to prevent in CI.

2. Score completeness before any draft request

Scoring is arithmetic over required keys, not a qualitative review of sentence quality or tone. A missing fixture hash is enough to mark PARTIAL even when both schemas are present and valid. Call the scorer in CI so a documentation job fails closed when tests have not recorded status codes. The command below prints the class and writes it back onto the packet for later steps.

#!/usr/bin/env python3
"""Worked example: score a docs evidence packet. Not a hosted service."""
from __future__ import annotations

import argparse
import hashlib
import json
from pathlib import Path

REQUIRED = (
    "operation_id",
    "source_sha",
    "request_schema",
    "response_schema",
    "fixture_path",
    "fixture_hash",
    "observed_status_codes",
)


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    digest.update(path.read_bytes())
    return "sha256:" + digest.hexdigest()


def score(packet: dict, repo: Path) -> tuple[str, list[str]]:
    gaps: list[str] = []
    for key in REQUIRED:
        if not packet.get(key):
            gaps.append(f"missing:{key}")

    fixture = repo / str(packet.get("fixture_path", ""))
    if packet.get("fixture_path") and not fixture.is_file():
        gaps.append("unreadable:fixture_path")
    elif fixture.is_file():
        actual = sha256_file(fixture)
        if actual != packet.get("fixture_hash"):
            gaps.append("mismatch:fixture_hash")

    codes = packet.get("observed_status_codes") or []
    if not isinstance(codes, list) or not codes:
        gaps.append("missing:observed_status_codes")

    if not packet.get("operation_id") and not packet.get("source_sha"):
        return "EMPTY", gaps
    if gaps:
        return "PARTIAL", gaps
    return "COMPLETE", []


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--packet", required=True)
    parser.add_argument("--repo", default=".")
    args = parser.parse_args()
    repo = Path(args.repo)
    path = Path(args.packet)
    packet = json.loads(path.read_text())
    klass, gaps = score(packet, repo)
    packet["completeness"] = klass
    packet["gaps"] = gaps
    path.write_text(json.dumps(packet, indent=2) + "\n")
    gap_dir = repo / "docs" / "gaps"
    gap_dir.mkdir(parents=True, exist_ok=True)
    op = packet.get("operation_id") or "unknown"
    (gap_dir / f"{op}.json").write_text(
        json.dumps({"completeness": klass, "gaps": gaps}, indent=2) + "\n"
    )
    print(f"{op}: {klass}")
    return 0 if klass == "COMPLETE" else 1


if __name__ == "__main__":
    raise SystemExit(main())

python scripts/score_packet.py --packet docs/packets/createCharge.json --repo .
# createCharge: PARTIAL

A PARTIAL result is a successful detection, not a weak draft that still deserves publication. The gap file is the artifact a reviewer should open before anyone asks a model for prose.

3. Emit gaps or abstain

PARTIAL packets should write a gap file under docs/gaps and exit non-zero in the documentation pipeline. EMPTY packets should skip the model entirely and open a ticket that assigns packet construction. COMPLETE packets are the only objects that may reach a draft template in this workflow. The rule is boring on purpose, because boredom is cheaper than reviewing invented status codes.

Map each gap string to an owner outside the model. missing:fixture_hash belongs to whoever maintains tests. missing:request_schema belongs to whoever owns the API contract. EMPTY belongs to whoever claimed a documentation page should exist for an operation that has no files. None of those owners is a completion endpoint.

4. Render from a locked template

Do not ask the model to choose headings, section order, or which status codes are worth mentioning. Supply a template whose placeholders map onto packet keys, then fill descriptions only from schema text already inside the packet. If a schema field has no description, the template must print TODO_HUMAN rather than a guessed sentence. Mechanical field tables should be rendered by the script; the model is optional restatement, not a table generator.

#!/usr/bin/env python3
"""Worked example: render a COMPLETE packet into a locked Markdown template."""
from __future__ import annotations

import argparse
import json
from pathlib import Path

TEMPLATE = """<!-- packet:{operation_id} sha:{source_sha} fixture:{fixture_hash} -->
# {operation_id}

## Request fields
{request_table}

## Response fields
{response_table}

## Observed status codes
{status_list}

## Example fixture
Bound to `{fixture_path}` (`{fixture_hash}`).
"""


def table(schema: dict) -> str:
    props = schema.get("properties") or {}
    if not props:
        return "TODO_HUMAN"
    lines = ["| Field | Type | Description |", "| --- | --- | --- |"]
    for name, spec in props.items():
        typ = spec.get("type", "object")
        desc = spec.get("description") or "TODO_HUMAN"
        lines.append(f"| `{name}` | `{typ}` | {desc} |")
    return "\n".join(lines)


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--packet", required=True)
    parser.add_argument("--out", required=True)
    args = parser.parse_args()
    packet = json.loads(Path(args.packet).read_text())
    if packet.get("completeness") != "COMPLETE":
        print("refuse: packet is not COMPLETE")
        return 1
    body = TEMPLATE.format(
        operation_id=packet["operation_id"],
        source_sha=packet["source_sha"],
        fixture_hash=packet["fixture_hash"],
        fixture_path=packet["fixture_path"],
        request_table=table(packet["request_schema"]),
        response_table=table(packet["response_schema"]),
        status_list="\n".join(
            f"- `{code}`" for code in packet["observed_status_codes"]
        ),
    )
    out = Path(args.out)
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(body + "\n")
    print(f"wrote {out}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

python scripts/render_reference.py \
  --packet docs/packets/createCharge.json \
  --out docs/reference/createCharge.md

Teams that already use MonkeyCode can run the scorer on the free server option without uploading the full repository. Free model access belongs after a COMPLETE score, and only to restate schema descriptions already stored in the packet. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The draft job should receive the packet path and locked template, not a request to improve the product story.

5. Lint the draft against the packet

After rendering, reject any status code, field name, or obligation verb that the packet does not authorize. Obligation verbs such as will, always, never, and guarantee remain human-owned even inside COMPLETE packets. The linter below is a worked example; extend the verb list until it matches your policy voice. A failed lint must fail the documentation job rather than open an optional review comment.

#!/usr/bin/env python3
"""Worked example: reject drafts that exceed the packet."""
from __future__ import annotations

import argparse
import json
import re
from pathlib import Path

OBLIGATION = re.compile(
    r"\b(will|always|never|guarantee|guaranteed|sla|backward compatible|supported until)\b",
    re.I,
)
STATUS = re.compile(r"\b([1-5][0-9]{2})\b")
PACKET_MARK = re.compile(
    r"<!--\s*packet:(?P<op>\S+)\s+sha:(?P<sha>\S+)\s+fixture:(?P<fx>\S+)\s*-->"
)


def lint(packet: dict, draft: str) -> list[str]:
    errors: list[str] = []
    mark = PACKET_MARK.search(draft)
    if not mark:
        errors.append("missing packet provenance comment")
    else:
        if mark.group("op") != packet.get("operation_id"):
            errors.append("provenance operation_id mismatch")
        if mark.group("sha") != packet.get("source_sha"):
            errors.append("provenance source_sha mismatch")
        if mark.group("fx") != packet.get("fixture_hash"):
            errors.append("provenance fixture_hash mismatch")

    allowed = {str(code) for code in packet.get("observed_status_codes", [])}
    for code in STATUS.findall(draft):
        if code not in allowed:
            errors.append(f"unauthorized status {code}")

    for match in OBLIGATION.finditer(draft):
        errors.append(f"human-owned obligation language: {match.group(0)}")

    if "TODO_HUMAN" in draft:
        errors.append("unfilled TODO_HUMAN remains in draft")
    return errors


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--packet", required=True)
    parser.add_argument("--draft", required=True)
    args = parser.parse_args()
    packet = json.loads(Path(args.packet).read_text())
    draft = Path(args.draft).read_text()
    errors = lint(packet, draft)
    for item in errors:
        print(item)
    return 1 if errors else 0


if __name__ == "__main__":
    raise SystemExit(main())

python scripts/lint_draft.py \
  --packet docs/packets/createCharge.json \
  --draft docs/reference/createCharge.md

Regenerating the same packet should not be the fix when the draft added a status code that tests never observed. The fix is a new fixture that records the code, or a human-authored page that owns the operational claim. That fork is the whole method: either evidence grows, or a human writes the sentence under a name.

Example CI sketch

The job below is an unexecuted sketch, not a report from a production pipeline. It exists to show fail-closed order: score, render, lint, and never call a model on PARTIAL input.

# Example only: unexecuted CI sketch for a docs compile job.
name: docs-packet
on: [push]
jobs:
  score-and-render:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python scripts/score_packet.py --packet docs/packets/createCharge.json
      - run: python scripts/render_reference.py --packet docs/packets/createCharge.json --out docs/reference/createCharge.md
      - run: python scripts/lint_draft.py --packet docs/packets/createCharge.json --draft docs/reference/createCharge.md

Wire the same commands locally before adding a remote runner. A packet that fails on a laptop will not become COMPLETE because the job moved to another machine.

What this does not cover

This workflow is the wrong tool for narrative tutorials, incident reviews, and any page that states a legal or commercial commitment. Do not use it when your API has no schema, no fixtures, and no recorded status codes from tests. Do not use it to draft migration guarantees, deprecation calendars, or regional data-handling promises. Those sentences need a human signer even when a model can imitate the house style with high fluency.

The packet also does not replace editorial review for diagrams, conceptual overviews, or onboarding sequences. Those documents synthesize across operations and therefore sit above a single-operation packet. Generate the reference leaves with this pipeline, then keep the synthesizing nodes on a human writing path. If your team publishes status pages or support windows, keep those files out of the renderer's output directory entirely.

Closing

Reference documentation becomes cheaper to trust when generation is refused more often than it is attempted. Score the packet, render only COMPLETE operations, and file every gap against a human owner. The Markdown that remains is a projection of files you already merge, not a second unsigned product story. Keep synthesizing tutorials and policy pages off this pipeline so the compile remains checkable in CI.