Freeze Tutorial Samples Before a Model Drafts the Walkthrough

Models may draft tutorial narration, but they must not invent the samples that the narration claims to demonstrate. A human-owned fixture ledger should freeze inputs, commands, and expected outcomes before any walkthrough prose is requested. The checker in this workflow rejects drafts that add paths, commands, or result strings missing from that ledger. This split keeps generated prose readable without letting it redefine what a reviewer can actually rerun.

Why invented samples break tutorials

Tutorial pages fail quietly when a plausible sentence outruns the sample that a reviewer can execute again. A model can name a friendly file, describe a clean import, and state a row count that never existed in the tree. Readers then copy the prose, miss the real fixture, and treat a narrative success as an observed result. The defect is evidentiary rather than stylistic, because the page no longer points at a frozen demonstration.

A ledger sets that evidence boundary before drafting starts, instead of hoping a late edit catches every invented detail. Each fixture carries an identifier, one command, input paths, and expected fragments that a person has already verified. Narration may cite those identifiers and restate those fragments, but it may not add neighbors that only sound consistent. The rule is narrow enough to automate and strict enough to change what reaches editorial review.

What the model may draft

The model may draft transitions that move readers from one approved fixture identifier to the next approved identifier. It may explain a human-written note already stored on the fixture, using that note as the factual ceiling. It may propose narration headings when those headings do not smuggle a new command, path, or outcome. It may not create examples, rename files, or rewrite a failed expectation as a successful run.

What a human must own

A human owner must freeze every command, input path, and expected fragment before the draft request goes out. That owner must also record the source revision that produced the expectation, so later readers can see the observed tree. Security cautions, supported versions, and any statement about duration or scale remain human-owned because a token checker cannot judge them. If nobody can rerun the fixture on the named revision, the page is not ready for model narration.

Workflow

1. Capture a fixture from a known tree

Run the tutorial command against a committed input, then save the exit status and the output fragments you will publish. Paste those fragments into the ledger instead of summarizing them in looser language that a model can stretch. Assign a stable fixture identifier that must appear in both the ledger file and the drafted page. Leave a field blank rather than guessing a revision, a filename, or an output fragment you did not observe.

Capture the source revision and the raw command output before anyone transcribes fragments into ledger fields. Redirect that output to a temporary file so the owned fragment is copied, not remembered from a scrollback line. Discard the temporary file after the ledger commit, because the ledger fragment is the published contract. The shell sequence below is a proposed capture ritual, and it assumes the ingest tool already exists in the tree.

git rev-parse HEAD
python3 tools/ingest.py --input fixtures/in/two-rows.csv > /tmp/fx-ingest-01.out
echo "exit_status=$?"

2. Store the ledger beside the page

Commit the ledger beside the tutorial so review diffs can separate evidence changes from narration changes. Treat every ledger edit as an evidence change that requires the owner, even when the prose diff looks small. Keep human notes short, concrete, and limited to claims that the same owner has already rerun. The sample ledger later in this page is a proposed shape, not a report from a measured production run.

3. Request narration that cites identifiers only

Send the ledger, the allowed section order, and a rule that every command must be copied in backticks exactly. Ask for prose that references fixture identifiers, and forbid new paths, counts, timings, or file names. Keep the model away from editing the ledger file, because drafting access is not ownership of evidence. If the draft needs a limit or a warning, that sentence must already exist in a human-written note.

4. Reject undeclared facts before editorial review

The checker extracts inline commands and compares them with the command strings stored on the ledger. It also flags backtick paths and quoted result strings that contain digits but do not appear in the ledger. A failing report names the line and the undeclared token, which is enough to return the draft unread. A passing report means only that the draft stayed inside the frozen set, not that the prose is wise.

5. Review causality that the checker cannot see

Read each paragraph for causes, recommendations, and scope claims that use allowed tokens and still mislead. The checker will not catch a sentence that blames the wrong subsystem while citing a valid fixture identifier. Approve the page only after a person confirms that each cited fixture still matches the named source revision. Store that review decision outside the model transcript so a later editor can see who accepted the causality.

Claim contract

The table below separates claim types that automation can fence from claim types that still need a person. Use it as a review contract when someone asks whether a generated sentence is allowed to ship. A row marked human-only should never be delegated merely because the checker returned a clean success code.

Claim	Owner	Model may	Checker result
Exact command	Human	Copy the stored string	Fail if the backtick text differs
Input path	Human	Cite a listed path	Fail on an unknown backtick path
Quoted output fragment	Human	Quote a listed fragment	Fail if digits were not stored
Fixture identifier	Human	Cite known identifiers	Fail if any owned id is missing
Cause, scale, or warning	Human only	Restate a human note only	Not detected; a person must decide

Proposed ledger

The following YAML is a template for a two-row ingest tutorial, and every value must be replaced after a real run. Do not publish the placeholder revision, and do not treat the fragment as evidence gathered for this article. The identifier FX-INGEST-01 is the only sample the narration in the good draft is allowed to cite. Human notes stay short so a drafting model cannot expand them into new guarantees or extra files.

# docs/ledgers/ingest-csv.yaml
# Proposed template. Replace every value after a reviewed local run.
ledger_version: 1
page_id: ingest-csv-walkthrough
source_revision: "REPLACE_WITH_GIT_REVISION"
fixtures:
  - id: FX-INGEST-01
    command: "python tools/ingest.py --input fixtures/in/two-rows.csv"
    inputs:
      - fixtures/in/two-rows.csv
    expect:
      exit_code: 0
      stdout_contains:
        - "rows=2"
    human_notes: "This fixture shows two data rows. Do not generalize the count."

Proposed checker

Place the checker under scripts, and keep it free of network calls so review does not depend on an external service. The implementation is an unexecuted proposal for local adaptation, not a certified linter with published accuracy figures. It reads a ledger and a markdown draft, then exits non-zero when the draft leaves the owned set. Reviewers should read every finding before they discuss tone, structure, or heading order on the page.

#!/usr/bin/env python3
"""Reject tutorial drafts that invent commands, paths, or quoted outcomes.

Proposed review tool. It does not execute fixtures or certify prose quality.
"""

from __future__ import annotations

import re
import sys
from pathlib import Path

import yaml

INLINE_CODE = re.compile(r"`([^`\n]+)`")
FIXTURE_ID = re.compile(r"\bFX-[A-Z0-9-]+\b")
QUOTED = re.compile(r'"([^"\n]+)"')


def load_ledger(path: Path) -> dict:
    data = yaml.safe_load(path.read_text(encoding="utf-8"))
    if not isinstance(data, dict) or not isinstance(data.get("fixtures"), list):
        raise SystemExit("ledger must contain a fixtures list")
    commands, paths, expected, ids = set(), set(), set(), set()
    for item in data["fixtures"]:
        ids.add(item["id"])
        commands.add(item["command"])
        paths.update(item.get("inputs") or [])
        expect = item.get("expect") or {}
        expected.update(expect.get("stdout_contains") or [])
        if "exit_code" in expect:
            expected.add(str(expect["exit_code"]))
    return {
        "commands": commands,
        "paths": paths,
        "expected": expected,
        "ids": ids,
    }


def lint(markdown: str, ledger: dict) -> list[str]:
    findings: list[str] = []
    for line_no, line in enumerate(markdown.splitlines(), start=1):
        for token in INLINE_CODE.findall(line):
            if token in ledger["commands"] or token in ledger["paths"]:
                continue
            if token.startswith("python ") or token.startswith("./"):
                findings.append(f"{line_no}: undeclared command: {token}")
            elif "/" in token:
                findings.append(f"{line_no}: undeclared path: {token}")
        for quoted in QUOTED.findall(line):
            if any(character.isdigit() for character in quoted):
                if quoted not in ledger["expected"]:
                    findings.append(
                        f"{line_no}: quoted result not in ledger: {quoted}"
                    )
    cited = set(FIXTURE_ID.findall(markdown))
    for missing in sorted(ledger["ids"] - cited):
        findings.append(f"draft never cites owned fixture {missing}")
    for unknown in sorted(cited - ledger["ids"]):
        findings.append(f"draft cites unknown fixture {unknown}")
    return findings


def main() -> int:
    if len(sys.argv) != 3:
        print(
            "usage: check_tutorial_ledger.py LEDGER.yaml DRAFT.md",
            file=sys.stderr,
        )
        return 2
    findings = lint(
        Path(sys.argv[2]).read_text(encoding="utf-8"),
        load_ledger(Path(sys.argv[1])),
    )
    if findings:
        print("\n".join(findings))
        return 1
    print("ledger-bound: draft stayed inside the owned fixture set")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Drafts the checker should separate

Save the failing draft as docs/drafts/bad-ingest.md and expect a non-zero exit, because the path, the command, and the quoted count are not owned. Save the passing draft as docs/drafts/good-ingest.md and expect exit zero only when the ledger text matches those backticks exactly. These pages are fixtures for the linter, not recommended reader-facing copy. Replace them when your repository uses different commands or different owned fragments.

Run `python tools/ingest.py --input fixtures/in/other.csv`.
The tool prints "rows=3" when the file is valid.

FX-INGEST-01 runs `python tools/ingest.py --input fixtures/in/two-rows.csv`.
The owned input is `fixtures/in/two-rows.csv`.
The reviewed fragment is "rows=2", and the owned exit status is "0".
This fixture shows two data rows. Do not generalize the count.

How to run the check

Run the checker from the repository root after you install PyYAML in the same environment that will perform review. The first command below should exit non-zero on a draft that invents a row count or an input path. The second command should exit zero only when every cited command, path, and quoted result is already owned. Treat both commands as part of the proposed workflow, and rerun them after any ledger edit.

python3 -m pip install pyyaml
python3 scripts/check_tutorial_ledger.py docs/ledgers/ingest-csv.yaml docs/drafts/bad-ingest.md
python3 scripts/check_tutorial_ledger.py docs/ledgers/ingest-csv.yaml docs/drafts/good-ingest.md

How to read a failure

A draft that says the command printed "rows=3" should fail when the ledger only contains the fragment "rows=2". A draft that introduces fixtures/in/other.csv should fail even when the surrounding paragraph cites FX-INGEST-01 correctly. A draft that never mentions FX-INGEST-01 should also fail, because owned fixtures are not optional decoration. These examples are illustrations of the rule, not logs from a shared benchmark or a customer incident.

Where drafting help fits

Once the ledger is the only factual attachment, MonkeyCode's free model access can draft narration without editing evidence files. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free server option matters when you want this same checker run outside one laptop, on a checkout you provision yourself. Neither availability claim adds model names, quotas, or uptime promises, and neither claim replaces the human owner.

Use the free access only after the ledger file is committed, so the draft prompt cannot outrun review history. Teams that already review fixtures can draft narration with that free model access, then run the checker before human causality review. Stop there, and do not treat a successful checker run as permission to skip the causality read.

Limitations

The checker misses paraphrases such as "about two rows" because it does not understand quantities written without the owned fragment. It also misses false causes, outdated advice, and confident tone wrapped around a legitimately cited fixture identifier. Number detection is deliberately narrow, so a digit hidden in a worded estimate will not raise a finding by itself. Passing therefore cannot prove that the fixture still runs, that the revision is current, or that the tutorial is complete.

Commands that do not start with python or ./ are ignored by the current command rule. Owners should keep published commands in those forms, or extend the prefixes before relying on the report. The script depends on PyYAML and on straight double quotes, so a smart quote around a result will slip past the scan. A clean exit is a fence around tokens, not a substitute for reading the tutorial as a newcomer would.

Who should not use this

Skip this workflow when no named person will own the ledger and refuse model edits to evidence fields. Skip it for incident runbooks, legal text, or security procedures where even allowed narration can leak sensitive context. Skip it when samples cannot be rerun, because a frozen fragment from an unknown tree is not ownership. Also skip it if the tutorial's value is exploratory opinion rather than a demonstration tied to committed inputs.

Closing rule

The practical conclusion is small: freeze the demonstration first, then allow narration to cite that demonstration and nothing beyond it. Generated tutorials stay reviewable when every sample outcome has an owner, an identifier, and a checker that can fail closed. If the ledger is empty, do not ask a model to invent the tutorial that the ledger was supposed to constrain.