Put Extracted Facts in JSON; Keep Product Claims in a Human File

Generated documentation fails when observed facts and product claims share one paragraph without a machine-readable boundary. A model can extract facts from a pinned checkout into JSON that tests can verify against the tree. A human must own claims such as support scope, breaking-change language, and compatibility promises, because those statements are not proven by source. The merge step belongs in CI, and it should refuse prose that smuggles a claim without a signed identifier.

This article proposes a two-ledger documentation workflow. It is a template you can run against a local repository, not a report of production metrics. The extract job is the only stage where a free model and a free server are useful. Everything after extraction is ordinary schema checks, string policy, and a renderer that cannot invent promises.

Mixed sentences are the defect, not weak tone

Most generation pipelines treat a Markdown file as the unit of work, which hides the cheaper failure. A fact is true or false against HEAD, such as a flag name, a default value, or a required environment variable. A claim is a promise about users, time, risk, or support, and the tree cannot confirm it. When those two statement types sit in one paragraph, reviewers argue about fluency instead of evidence.

Cheap model output makes that mix cheaper to produce and slower to catch. The page still reads like documentation, so the defect survives grammar review. The corrective split is mechanical: facts become rows, claims become signed records, and public prose is a template that may only interpolate both under CI.

Two ledgers and one published page

Keep three artifacts in the repository, and treat only the third as public documentation.

1. facts/*.json — model-draftable, schema-checked, regenerated from a pinned commit.
2. claims.yaml — human-owned, never overwritten by a model, reviewed like shipped code.
3. docs/**/*.md — rendered output from a human template that interpolates facts and named claims.

The model may fill the fact ledger from listed source paths. The model may not create claim identifiers, and it may not rewrite claim text. The human template is the only place a claim identifier may appear next to a fact field.

Artifact: schema, allowlist, and a merge gate

The files below are a proposed local template. They are not executed against a public corpus in this article, and they do not assert latency or quality scores.

1. Constrain what a fact is allowed to say

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "DocFactLedger",
  "type": "object",
  "additionalProperties": false,
  "required": ["source_commit", "source_paths", "facts"],
  "properties": {
    "source_commit": { "type": "string", "minLength": 7 },
    "source_paths": {
      "type": "array",
      "minItems": 1,
      "items": { "type": "string" }
    },
    "facts": {
      "type": "array",
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["id", "kind", "symbol", "value", "evidence"],
        "properties": {
          "id": { "type": "string", "pattern": "^[a-z0-9_]+$" },
          "kind": {
            "type": "string",
            "enum": ["flag", "default", "env", "endpoint", "error_code"]
          },
          "symbol": { "type": "string", "minLength": 1 },
          "value": { "type": "string", "minLength": 1 },
          "evidence": {
            "type": "object",
            "additionalProperties": false,
            "required": ["path", "start_line", "end_line"],
            "properties": {
              "path": { "type": "string" },
              "start_line": { "type": "integer", "minimum": 1 },
              "end_line": { "type": "integer", "minimum": 1 }
            }
          }
        }
      }
    }
  }
}

The schema refuses free-form description fields on purpose. If the extractor needs a comment, that comment belongs in a review note, not in published documentation. Facts without line evidence are invalid, which keeps the model from inventing symbols that do not exist in the tree.

2. Keep claims in a file no model may write

# claims.yaml — human-owned. Models must not create or edit records.
claims:
  - id: CLAIM_SUPPORT_SCOPE
    text: "Community support covers the documented CLI surface only."
    owners: ["docs-oncall"]
    depends_on_facts: ["cli_help_flag", "cli_config_env"]
    signed_commit: "REPLACE_WITH_COMMIT"
  - id: CLAIM_BREAKING_WINDOW
    text: "Flag removals are breaking and require a major version."
    owners: ["api-steward"]
    depends_on_facts: ["cli_help_flag"]
    signed_commit: "REPLACE_WITH_COMMIT"

Each claim has an owner and a fact dependency list. When a depended-on fact changes at HEAD, CI should fail until a human updates signed_commit. That failure is the review signal; a regenerated paragraph is not.

3. Detect claim language smuggled into fact values

# tools/check_fact_ledger.py
from __future__ import annotations

import json
import re
import sys
from pathlib import Path

CLAIMISH = re.compile(
    r"\b(guarantee|sla|supported|unsupported|never break|will not|"
    r"production-ready|enterprise|customers? must|we promise|certified)\b",
    re.I,
)

FORBIDDEN_KIND_CROSSING = {
    "error_code": re.compile(r"\b(user|customer|you must)\b", re.I),
}

def load_ledger(path: Path) -> dict:
    data = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(data.get("facts"), list):
        raise SystemExit(f"{path}: facts must be a list")
    return data

def check_file(path: Path) -> list[str]:
    errors: list[str] = []
    ledger = load_ledger(path)
    for fact in ledger["facts"]:
        blob = f"{fact['symbol']} {fact['value']}"
        if CLAIMISH.search(blob):
            errors.append(f"{path}:{fact['id']}: claim-like language in a fact value")
        extra = FORBIDDEN_KIND_CROSSING.get(fact["kind"])
        if extra and extra.search(blob):
            errors.append(f"{path}:{fact['id']}: kind {fact['kind']} may not address users")
        evidence = fact["evidence"]
        if evidence["end_line"] < evidence["start_line"]:
            errors.append(f"{path}:{fact['id']}: evidence line range is inverted")
    return errors

def main() -> int:
    root = Path("facts")
    if not root.exists():
        print("facts/ is missing", file=sys.stderr)
        return 1
    errors: list[str] = []
    for path in sorted(root.glob("*.json")):
        errors.extend(check_file(path))
    for item in errors:
        print(item, file=sys.stderr)
    return 1 if errors else 0

if __name__ == "__main__":
    raise SystemExit(main())

This check is deliberately conservative. It will block some legitimate values, and that is cheaper than publishing an accidental support promise inside a default-value sentence.

4. Merge only through a human template

# tools/merge_docs.py
from __future__ import annotations

import json
import re
import sys
from pathlib import Path

import yaml

FACT_TOKEN = re.compile(r"\{\{fact\.([a-z0-9_]+)\.value\}\}")
CLAIM_TOKEN = re.compile(r"\{\{claim\.([A-Z0-9_]+)\.text\}\}")
BARE_PROMISE = re.compile(
    r"\b(guarantee|we promise|supported in production)\b", re.I
)

def index_facts(ledger: dict) -> dict:
    return {row["id"]: row for row in ledger["facts"]}

def index_claims(payload: dict) -> dict:
    return {row["id"]: row for row in payload["claims"]}

def render(template: str, facts: dict, claims: dict) -> str:
    def fact_sub(match: re.Match[str]) -> str:
        key = match.group(1)
        if key not in facts:
            raise SystemExit(f"unknown fact id: {key}")
        return facts[key]["value"]

    def claim_sub(match: re.Match[str]) -> str:
        key = match.group(1)
        if key not in claims:
            raise SystemExit(f"unknown claim id: {key}")
        return claims[key]["text"]

    page = FACT_TOKEN.sub(fact_sub, template)
    page = CLAIM_TOKEN.sub(claim_sub, page)
    if BARE_PROMISE.search(page):
        raise SystemExit("rendered page contains unsigned claim language")
    leftover = re.search(r"\{\{[^}]+\}\}", page)
    if leftover:
        raise SystemExit(f"unresolved token: {leftover.group(0)}")
    return page

def main() -> int:
    facts = index_facts(json.loads(Path("facts/cli.json").read_text(encoding="utf-8")))
    claims = index_claims(yaml.safe_load(Path("claims.yaml").read_text(encoding="utf-8")))
    template = Path("templates/cli.md.in").read_text(encoding="utf-8")
    Path("docs/cli.md").write_text(render(template, facts, claims), encoding="utf-8")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

A valid template talks to identifiers, not to the model. Example input:

# CLI reference

The help flag is `{{fact.cli_help_flag.value}}`.
Configuration is read from `{{fact.cli_config_env.value}}`.

{{claim.CLAIM_SUPPORT_SCOPE.text}}
{{claim.CLAIM_BREAKING_WINDOW.text}}

If a generated fact file tries to include support language, the fact linter fails before merge. If a template author writes a promise without a claim token, the renderer fails after merge. Those two failures are different owners, and they should not be collapsed into one review comment.

Numbered workflow for a command-reference page

Use one narrow page first. A CLI reference is a better proving ground than a product overview, because most of its sentences should be facts.

1. Pin the checkout with git rev-parse HEAD and record that digest in the ledger filename or payload.
2. List extractable paths in a manifest such as docgen/source_paths.txt, and refuse paths outside that list.
3. Run the extract job against those paths only, writing facts/cli.json and nothing else.
4. Validate JSON Schema, evidence ranges, and the claim-language regex before any Markdown exists.
5. Confirm every depends_on_facts entry still matches current fact identifiers and values.
6. Render templates/cli.md.in to docs/cli.md in CI, then fail the job on unresolved tokens.
7. Require a human signature on claims.yaml when a depended-on fact hash changes.

The extract job is the stage where hosted model access matters. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can run that extract job against a checkout without turning the whole handbook into a chat transcript. The rest of the gate is local Python, schema, and git policy, and it does not depend on a named model, quota, or hardware profile.

A minimal job wrapper can look like this proposed shell, with the endpoint supplied by the environment rather than committed:

#!/bin/sh
set -eu
COMMIT=$(git rev-parse HEAD)
PATHS=$(tr '\n' ' ' < docgen/source_paths.txt)

python tools/extract_facts.py \
  --commit "$COMMIT" \
  --paths $PATHS \
  --out facts/cli.json

python tools/check_fact_ledger.py
python tools/merge_docs.py
git diff --exit-code -- docs/cli.md claims.yaml || true

Keep secrets out of the prompt body. The extractor should receive file slices and a JSON schema, not a request to "write friendly docs for customers." Friendly prose is how claims enter a fact field.

What the model may draft versus what a human must own

The allowlist is easier to maintain when it is expressed as kinds, not as vibes.

Kind	Model may draft	Human must own
Flag, default, env, endpoint, error code	Symbol, value, evidence range	Whether the symbol is supported for customers
Command help text copied from --help	Verbatim extract with source lines	Compatibility promises across versions
Example invocation	A command that matches current flags	That the example is covered by support
Release note bullet from a commit subject	Observed subject and hash	Breaking versus non-breaking classification
Runbook recovery step	Commands present in scripts	When to declare an incident, and who is on call

If a sentence cannot be typed as one of the fact kinds, it is a claim or it does not belong in the generated page. "Should," "must" directed at customers, time bounds, and support verbs are claims even when they appear inside an example note.

Tests that should fail on purpose

# tools/test_doc_gate.py
from check_fact_ledger import check_file
from pathlib import Path
import json
import tempfile

SAMPLE = {
    "source_commit": "abc1234",
    "source_paths": ["cmd/root.go"],
    "facts": [{
        "id": "cli_help_flag",
        "kind": "flag",
        "symbol": "--help",
        "value": "supported for production customers",
        "evidence": {"path": "cmd/root.go", "start_line": 4, "end_line": 6}
    }]
}

def test_claim_language_in_fact_value_is_rejected(tmp_path: Path) -> None:
    path = tmp_path / "cli.json"
    path.write_text(json.dumps(SAMPLE), encoding="utf-8")
    errors = check_file(path)
    assert errors, "smuggled support language must fail the fact ledger"

Add a second test that a template without {{claim.*}} cannot contain the same support sentence. The two tests encode the ownership split: extractors fail on claims, templates fail on unsigned promises. If both tests are skipped, the workflow is only a formatter.

Limitations and who should not use this

This approach assumes documentation has a repository, a CI system, and a named owner for claims. Teams that publish docs only from a CMS, or that treat the public page as the contract itself, will not get a clean split. Legal terms, security advisories, pricing, availability, and regulated instructions should stay out of the fact ledger even when a model could paraphrase them.

The regex will both overblock and underblock. It does not understand negation, quotations, or sarcasm, and it will miss novel promise phrasing. Schema drift is another real cost: when engineers add a new flag kind, someone must extend the enum before extraction can continue. The workflow also does not prove that evidence lines still say what the fact value claims; a later pass can hash the slice, but that check is not in the template above.

Do not use this method to generate incident blame, customer-facing root-cause language, or migration verdicts. Those are claims with operational consequences, and a fluent extract does not reduce that ownership. Do not use it to fill empty handbooks in one pass. The first successful run should be one command-reference page with fewer than a few dozen facts.

If you already have free model access, run the extract job on a single reference page and keep the claim file in human review rather than expanding the prompt.