Run the Fence Before You Ship the Paragraph: An Executable-Example Gate for Model-Drafted Docs

A model may draft the prose of a documentation page, but it may not own the example that proves the prose. That single rule converts doc generation from a trust argument into a build step with an exit code, and it holds even when the drafting model is free, fast, and only loosely supervised. Everything below is a working gate built on that rule: prose is model-draftable, executed code is the evidence, and any example that cannot run becomes a named human's problem.

The failure mode this targets is not bad writing. Documentation pages usually rot at the example rather than the sentence: a flag gets renamed, a response field moves, a command silently starts printing a warning that breaks a copy-paste pipeline. Prose drifts slowly and reads plausibly the whole way down, while a broken fence fails loudly the moment somebody executes it. So the gate asks the cheap, objective question first — does this block still run? — and lets that answer decide how much draft text is allowed to survive.

Step 1: Declare the ownership boundary before drafting

Write the boundary down as a one-line policy, because models follow explicit scopes far more reliably than implicit ones. The model owns sentences that describe behaviour, rationale, and trade-offs; the human owns every fenced example and every sentence whose only support is a failing fence. This split matters because it maps to different verification costs: prose is reviewed by reading, examples are reviewed by executing.

A page therefore has two artifacts with two owners. The prose artifact can be regenerated as often as you like, and the example artifact changes only when a human accepts a new execution result. Keep that distinction visible in your repo layout, so reviewers never have to guess which part of a diff they are approving.

Step 2: Extract and execute every fence

The gate itself is deliberately boring: walk the docs tree, pull out fenced blocks, run the ones that have a runner, and emit a JSON report. Info strings carry control flags, so a block written as `python noexec is parsed but never executed.

`python

!/usr/bin/env python3

"""tools/fence_gate.py - execute every runnable fence under a docs tree."""
from future import annotations

import json
import shutil
import subprocess
import sys
import tempfile
from dataclasses import dataclass
from pathlib import Path

RUNNERS = {
"python": (".py", ["python3", "{file}"]),
"py": (".py", ["python3", "{file}"]),
"bash": (".sh", ["bash", "{file}"]),
"sh": (".sh", ["bash", "{file}"]),
"js": (".mjs", ["node", "{file}"]),
"javascript": (".mjs", ["node", "{file}"]),
}

@dataclass
class Fence:
doc: str
line: int
lang: str
flags: list[str]
body: str
status: str = "skipped" # passed | failed | skipped | unowned
detail: str = ""

def parse_fences(doc: Path) -> list[Fence]:
fences: list[Fence] = []
inside, start, info, body = False, 0, "", []
for lineno, line in enumerate(doc.read_text(encoding="utf-8").splitlines(), start=1):
stripped = line.strip()
if not inside and stripped.startswith(""):
inside, start, info, body = True, lineno, stripped[3:].strip(), []
continue
if inside and stripped.startswith(""):
words = info.split()
lang = words[0].lower() if words else ""
fences.append(Fence(str(doc), start, lang, words[1:], "\n".join(body)))
inside = False
continue
if inside:
body.append(line)
return fences

def run_fence(fence: Fence, timeout: int = 20) -> None:
if "noexec" in fence.flags:
fence.status, fence.detail = "unowned", "marked noexec: needs a named owner"
return
runner = RUNNERS.get(fence.lang)
if runner is None:
fence.status, fence.detail = "skipped", f"no runner for {fence.lang!r}"
return
suffix, argv_tpl = runner
exe = shutil.which(argv_tpl[0])
if exe is None:
fence.status, fence.detail = "skipped", f"{argv_tpl[0]} not installed"
return
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / f"fence{suffix}"
path.write_text(fence.body, encoding="utf-8")
argv = [exe if a == argv_tpl[0] else a.replace("{file}", str(path)) for a in argv_tpl]
try:
proc = subprocess.run(argv, cwd=tmp, capture_output=True, text=True, timeout=timeout)
except subprocess.TimeoutExpired:
fence.status, fence.detail = "failed", f"timeout after {timeout}s"
return
fence.status = "passed" if proc.returncode == 0 else "failed"
if proc.returncode != 0:
tail = proc.stderr.strip().splitlines()
fence.detail = tail[-1][:200] if tail else f"exit {proc.returncode}"

def main(root: str, out: str) -> int:
results: list[Fence] = []
for doc in sorted(Path(root).rglob("*.md")):
for fence in parse_fences(doc):
run_fence(fence)
results.append(fence)
Path(out).write_text(json.dumps([f.dict for f in results], indent=2), encoding="utf-8")
bad = [f for f in results if f.status in {"failed", "unowned"}]
for f in bad:
print(f"{f.status:8} {f.doc}:{f.line} [{f.lang}] {f.detail}")
return 1 if bad else 0

if name == "main":
raise SystemExit(main(sys.argv[1] if len(sys.argv) > 1 else "docs",
sys.argv[2] if len(sys.argv) > 2 else "fence-report.json"))
`

Run it as python3 tools/fence_gate.py docs fence-report.json. The non-zero exit on failure is the whole point: the gate is a test, not a report you read when convenient. Treat a red gate exactly like a red unit test suite.

Step 3: Turn gate results into a routing decision

Four outcomes are enough to cover real docs, and each one maps to a different owner. Encode the mapping once so nobody relitigates it in review.

Gate result	Who owns the paragraph above the fence	Ship the draft?
passed	Model draft allowed	Yes, review as prose
failed	Named human in owners.yml	No — revert to last shipped text
skipped (no runner installed)	Named human in owners.yml	Yes, marked unverified in front matter
unowned (noexec)	Named human must claim the block	No, until claimed

The skipped row is where most teams quietly cheat. A missing language runtime is an environment fact, not a documentation fact, so the honest response is to keep the example, label it unverified, and assign a person — not to delete it and pretend the page is fully checked.

Step 4: Route failures to a name, not a channel

Ownership that lives in a shared inbox is ownership that nobody holds. Keep a small map from doc path to a person, and let the gate generate the work items directly from the JSON report.

`yaml

docs/owners.yml - routing targets for fences the gate cannot clear

"guide/install.md": "@avery"
"guide/webhooks.md": "@ops-docs"
"reference/cli.md": "@avery"
default: "@docs-oncall"
`

`bash
jq -r '.[] | select(.status == "failed" or .status == "unowned")
| "- [ ] `\(.doc):\(.line)` [\(.status)] — \(.detail)"' \
fence-report.json > fence-issues.md
`

That fence-issues.md is the human queue, and it is intentionally short. A page with nine broken examples produces nine checkboxes with file and line numbers, which is a far better review artifact than a paragraph explaining that the docs "may be slightly outdated".

Step 5: Let the free model draft inside the evidence boundary

This is where I actually use MonkeyCode: its free model access handles the prose pass, and the free server option runs the fence sweep on a schedule so drift shows up even when my laptop is closed. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Availability and limits are operator-supplied and change over time, so verify them against the product's current documentation before you build a pipeline around them.

The drafting prompt gets two inputs — the current page and the fence report — and one hard constraint: it may only describe commands, flags, or output shapes that appear in a passed fence. Anything else must be written as an open question addressed to the owner in owners.yml. In practice the model becomes a rewriter of verified material rather than an author of plausible material, and the questions it emits are often the most useful part of the diff.

Schedule the sweep however your environment already schedules jobs, for example 0 6 * * 1 python3 tools/fence_gate.py docs fence-report.json for a Monday morning pass. If you run it on the free server, the command does not change — only the machine that executes it does, which is exactly what you want from a gate.

Limitations and who should skip this

The gate only proves that a block exits zero, never that it demonstrates the right thing; a fence can pass while teaching an obsolete workflow. Examples that need credentials, network access, or destructive side effects will either fail or need a sandbox you maintain, and flaky ones will teach your team to ignore red builds. Languages without a cheap local runner produce skipped rows forever unless somebody writes a runner.

Skip this approach if your docs are mostly conceptual, if you have no CI or scheduled jobs to hang it on, or if nobody is willing to be the name in owners.yml. A gate with no owner is just a slower way to generate unflagged warnings. Start with one high-traffic page, run the gate for two weeks, and only then decide whether the prose pass is worth automating across the tree.

If you try it, the smallest useful version is Step 2 plus a single owner entry — that combination already tells you which half of your documentation is actually verified.