Bind Generated Doc Blocks to a Command, Schema, or File

Documentation generation fails most often when a model writes claims that no test, schema, or log can falsify. Reviewers then debate tone while an unverifiable sentence ships into a page that operators will treat as truth. The practical fix is not a better prompt; it is a contract that binds every drafted block to a verification source. If that source is mechanical, missing, or explicitly human, the workflow already decides who may write the next sentence.

Unverifiable claims are a process bug

A generated paragraph can read fluently and still be wrong in a way that ordinary diff review cannot catch. Restarting a worker is observable when a status command exists; meeting a production SLA is a commitment no model should author. File-level allowlists make the grain too coarse, because one markdown page mixes commands, examples, policy, and support promises. Permission must attach to the block, and evidence must attach to a verification source the validator can resolve.

Generated pages rot faster than handwritten ones when the prose has no parent artifact to go stale against. Ordinary grep will not catch a promise that was never backed by a ticket, schema, or recorded command. Treat that mismatch as a process bug, not as a writing-quality issue for the model.

Claim classes are the policy, not the prompt

Use a closed vocabulary and refuse unknown labels at validation time. The table below is the entire authorization model for drafting. Prompts do not get to invent extra classes, and a missing verify_from object is a failed build rather than a reviewer comment.

claim_class	Model may draft	Required verify_from.kind	Human must own
observable	yes	schema, file, or command	the choice of source
derived_example	yes, labeled as example	schema or file	whether the example is canonical
procedure	yes, after a recorded run	command plus stdout hash	whether the run represents production
rationale	no	none	the tradeoff and the date it was accepted
commitment	no	none	SLA, support, pricing, legal, GA status

The hashes in later examples are placeholders for a recorded run, not measurements from a production host. Replace them before enabling --replay or --strict on a real tree.

1. Split each page into labeled blocks

Do not send a whole chapter to a model and hope that headings imply ownership. Split the page into comment-wrapped blocks that carry JSON metadata the validator can parse without a markdown dialect. Each block needs a stable id, a claim class, a draft flag, and a verification source. Keep stitching sentences outside the blocks when those sentences mix policy with examples.

---
page_id: workers.restart
owner: platform-docs
---

<!-- doc-block
{
  "id": "workers.restart.observable.status",
  "claim_class": "observable",
  "model_drafted": false,
  "verify_from": {
    "kind": "command",
    "argv": ["python", "scripts/workerctl.py", "status"],
    "stdout_sha256": "REPLACE_WITH_RECORDED_SHA256"
  }
}
-->
The status command prints `state=idle` when no job is leased.

<!-- doc-block
{
  "id": "workers.restart.procedure.dry_run",
  "claim_class": "procedure",
  "model_drafted": false,
  "verify_from": {
    "kind": "command",
    "argv": ["python", "scripts/workerctl.py", "restart", "--dry-run"],
    "stdout_sha256": "REPLACE_WITH_RECORDED_SHA256"
  }
}
-->
A dry-run restart prints the target pid and exits zero without signaling the process.

<!-- doc-block
{
  "id": "workers.restart.commitment.sla",
  "claim_class": "commitment",
  "model_drafted": false,
  "human_owner": "sre-oncall",
  "verify_from": {"kind": "none"}
}
-->
Restart is a best-effort operator action; it is not covered by the availability SLA.

Store command argv as an array, not as a shell string, so the replay path cannot pick up unexpected interpolation. Schema sources should point at a path plus a JSON pointer, and file sources should point at a glob that already exists in the repository.

2. Validate blocks before any draft call

The validator is the gate, and it must fail closed. Missing metadata, an unknown claim class, a commitment marked model_drafted, or a command block without a hash should fail the job. The script below is a runnable starting point; treat the sample hashes as placeholders until you record output on an intended host.

#!/usr/bin/env python3
"""Validate documentation blocks before a model is allowed to draft."""
from __future__ import annotations

import argparse
import hashlib
import json
import re
import subprocess
import sys
from pathlib import Path

BLOCK_RE = re.compile(r"<!--\s*doc-block\s*(?P<body>.*?)\s*-->", re.DOTALL)
ALLOWED_CLASSES = {
    "observable",
    "derived_example",
    "procedure",
    "rationale",
    "commitment",
}
DRAFTABLE = {"observable", "derived_example", "procedure"}
MECHANICAL = {"schema", "file", "command"}


def parse_blocks(text: str, source: Path) -> list[dict]:
    blocks = []
    for match in BLOCK_RE.finditer(text):
        try:
            payload = json.loads(match.group("body"))
        except json.JSONDecodeError as exc:
            raise ValueError(f"{source}: invalid JSON in doc-block ({exc})") from exc
        payload["_source"] = str(source)
        payload["_body_after"] = text[match.end():].split("<!--", 1)[0].strip()
        blocks.append(payload)
    return blocks


def validate_block(block: dict, strict: bool) -> list[str]:
    errors: list[str] = []
    ident = block.get("id", "<missing-id>")
    claim = block.get("claim_class")
    verify = block.get("verify_from") or {}
    kind = verify.get("kind")

    if claim not in ALLOWED_CLASSES:
        errors.append(f"{ident}: unknown claim_class {claim!r}")
    if not isinstance(verify, dict) or kind is None:
        errors.append(f"{ident}: verify_from.kind is required")
        return errors

    if block.get("model_drafted") and claim not in DRAFTABLE:
        errors.append(f"{ident}: {claim} cannot be model_drafted")

    if claim in {"rationale", "commitment"}:
        if kind != "none":
            errors.append(f"{ident}: {claim} requires verify_from.kind=none")
        if not block.get("human_owner"):
            errors.append(f"{ident}: {claim} requires human_owner")
    elif kind not in MECHANICAL:
        errors.append(f"{ident}: {claim} requires a mechanical verify_from.kind")

    if kind == "command":
        argv = verify.get("argv") or []
        digest = verify.get("stdout_sha256", "")
        if not argv or not all(isinstance(part, str) for part in argv):
            errors.append(f"{ident}: command argv must be a list of strings")
        if strict and not re.fullmatch(r"[0-9a-f]{64}", digest or ""):
            errors.append(f"{ident}: stdout_sha256 must be a recorded sha256")

    if kind == "file" and not verify.get("glob"):
        errors.append(f"{ident}: file source requires glob")
    if kind == "schema" and not (verify.get("path") and verify.get("pointer")):
        errors.append(f"{ident}: schema source requires path and pointer")
    if not block.get("_body_after"):
        errors.append(f"{ident}: block has no prose to verify")
    return errors


def replay_command(block: dict) -> str | None:
    verify = block["verify_from"]
    if verify.get("kind") != "command":
        return None
    completed = subprocess.run(
        verify["argv"],
        check=False,
        capture_output=True,
        text=True,
    )
    digest = hashlib.sha256(completed.stdout.encode("utf-8")).hexdigest()
    expected = verify.get("stdout_sha256")
    if completed.returncode != 0:
        return f"{block['id']}: command exited {completed.returncode}"
    if expected and expected != digest:
        return f"{block['id']}: stdout hash {digest} != {expected}"
    return None


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("root", type=Path)
    parser.add_argument("--strict", action="store_true")
    parser.add_argument("--replay", action="store_true")
    args = parser.parse_args()

    errors: list[str] = []
    seen: set[str] = set()
    for path in sorted(args.root.rglob("*.md")):
        for block in parse_blocks(path.read_text(encoding="utf-8"), path):
            ident = block.get("id")
            if not ident or ident in seen:
                errors.append(f"{path}: missing or duplicate id {ident!r}")
            seen.add(ident or "")
            errors.extend(validate_block(block, strict=args.strict))
            if args.replay:
                replay_error = replay_command(block)
                if replay_error:
                    errors.append(replay_error)

    for item in errors:
        print(item, file=sys.stderr)
    print(f"checked_blocks={len(seen)} errors={len(errors)}")
    return 1 if errors else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it against a docs directory before any model call, and keep --strict off until placeholder hashes are replaced.

python scripts/verify_docs.py docs
python scripts/verify_docs.py docs --strict
python scripts/verify_docs.py docs --strict --replay

3. Draft only blocks the validator already accepted

Build a payload from blocks whose claim class is draftable and whose source already validates. Send the id, the claim class, the verification source, and any recorded stdout or schema excerpt, not the rest of the handbook. Ask the model to return the same id and a prose field, then write that prose back under the original comment. If the model emits a new block, a new class, or a commitment, discard the response and fail the job.

A minimal request body can look like the following example, which is a contract rather than a chat transcript. Keep the instruction short so the validator remains the authority.

{
  "task": "draft_prose",
  "rules": [
    "Use only the supplied verify_from evidence.",
    "Do not add claim_class, SLA, support, or version promises.",
    "Label derived_example prose as an example in the first sentence."
  ],
  "block": {
    "id": "workers.restart.observable.status",
    "claim_class": "observable",
    "verify_from": {
      "kind": "command",
      "argv": ["python", "scripts/workerctl.py", "status"],
      "stdout": "state=idle\\n"
    }
  }
}

After the write, run the validator again so a dropped comment or a flipped model_drafted flag cannot land. The second pass is the actual merge gate; the model output is only a candidate patch.

4. Replay mechanical sources on a schedule

Command-backed blocks go stale when the binary changes and the sentence does not. Replay is the check that the recorded hash still matches stdout on an intended runner. Schema-backed blocks should fail when the pointer disappears, and file-backed blocks should fail when the glob matches zero paths. None of those failures require a reviewer to notice tone drift first.

python scripts/workerctl.py status | sha256sum
python scripts/verify_docs.py docs --strict --replay

Record hashes from the same argv the document claims, including flags. A dry-run restart hash must not be reused for a live restart block, because the procedure class is only as honest as the recorded command. If replay cannot run in the merge pipeline, run it on a scheduled job and open a docs-only change when hashes diverge.

5. Leave rationale and commitment blocks on a human patch

Human-owned blocks still need metadata, because silence is how unverifiable sentences re-enter the file. The model never drafts them, and the validator never treats kind: none as optional. Support hours, GA language, pricing, and availability promises stay on a named human_owner even when nearby observable blocks are regenerated every week.

When a page needs both a command excerpt and an SLA caveat, keep them in separate blocks. Reviewers then spend time on the commitment, not on whether the status sentence still matches stdout. That split is the whole point of binding verification sources at block grain rather than at file grain.

Teams that already run a local validator can still send only allowed blocks to a remote draft step. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option fit this loop because the draft pass and the replay job can share one environment without adding a separate docs toolchain. The workflow does not depend on a model name, a quota, or a hardware profile; it depends on refusing unbound blocks before prose is requested.

What to do when validation fails

Number the failure against the metadata, not against the model's confidence. The following order keeps the debug path mechanical.

1. If JSON does not parse, repair the comment before rereading the prose.
2. If claim_class is unknown, stop generating and extend the table only with a human change.
3. If a draftable block lacks a mechanical source, record a command, file, or schema pointer first.
4. If --replay hashes diverge, update the sentence from new stdout or revert the binary change.
5. If model_drafted is true on a commitment, delete the generated sentence and restore the human block.

A useful regression test is small and does not need a model in the loop. The cases below lock the fail-closed behavior so a later prompt change cannot widen the allowlist by accident.

from pathlib import Path
from verify_docs import parse_blocks, validate_block

SAMPLE = '''<!-- doc-block
{
  "id": "api.timeout.commitment",
  "claim_class": "commitment",
  "model_drafted": true,
  "verify_from": {"kind": "none"}
}
-->
Timeouts are covered by the gold SLA.
'''

def test_commitment_cannot_be_model_drafted(tmp_path: Path) -> None:
    path = tmp_path / "sla.md"
    path.write_text(SAMPLE, encoding="utf-8")
    block = parse_blocks(path.read_text(encoding="utf-8"), path)[0]
    errors = validate_block(block, strict=True)
    assert any("cannot be model_drafted" in item for item in errors)

Limitations

This contract does not prove that a recorded command represents production, only that stdout still matches a hash. A dry-run can be replayable and still be the wrong procedure for a live incident. Schema pointers do not catch semantic drift when a field remains present but changes meaning. The validator also cannot stop a human from pasting a commitment into an observable block under a mechanical source; review still has to read claim_class.

Placeholder hashes are a deliberate weak mode for bootstrapping. Leaving --strict off in merge pipelines will accept REPLACE_WITH_RECORDED_SHA256 and quietly skip the evidence rule. Replay on an unpinned runner can also produce hash thrash that looks like a docs failure when the real issue is environment drift.

Who should not use this approach

Skip the workflow for one-off README files that will never be regenerated, because the metadata cost exceeds the review cost. Skip it for purely legal or marketing pages where every sentence is already a commitment and no mechanical source exists. Skip it when the team cannot record command output on a stable runner, because replay will become theatre. Skip it when docs are generated from a single source such as OpenAPI with no handwritten commitments, and use schema generation instead of a model draft loop.

Start with one runbook and the validator in --strict mode; expand the allowlist only after replay stays green. The pages that improve first are operator procedures with commands you can already run twice, not the handbook chapters that exist to make promises.