Prove Every README Command Before You Let a Model Rewrite It

The cheapest way to lie to a new teammate is an old README. If a model rewrites that file before you prove which commands still exist, you just shipped a more confident lie.

I keep cloning repos where npm run dev is gone, make bootstrap never existed, and port 3000 is now 8088. Sound familiar? AI made the next patch cheap. It did not make the getting-started guide true.

This tutorial is a from-zero audit. Extract claims. Inventory the repo. Diff them. Only then — optionally — ask a free model to draft replacements for claims you already proved false. Every stage has a verification step. Skip the model and the article still works.

What you will have at the end

A small Python tool, readme_drift.py, plus a fixture repo you can run locally. It writes drift_report.json. That file is the artifact. The model is a guest, not the source of truth.

You need Python 3.11+, a git checkout you are allowed to scan, and a README that actually contains fenced shell blocks. No cloud account is required for stages 1–3.

Stage 1 — Extract the commands the README promises

Do not start with a prompt. Start with a parser. Why trust a paragraph when the dangerous bits live in fences?

Create a working directory and a deliberately stale fixture:

mkdir -p readme-drift/fixture && cd readme-drift
cat > fixture/README.md << 'EOF'
# Demo service

## Run it

bash
npm run dev
make bootstrap
pytest -q
curl http://localhost:3000/health

Set `DATABASE_URL` and `REDIS_URL` before you start.
EOF

python

Now the extractor. It only looks at fenced bash / sh / shell blocks and a short list of env-ish tokens. That is on purpose. Prose is where models hallucinate.

# extract.py
from __future__ import annotations

import re
from pathlib import Path

FENCE = re.compile(r"```

(?:bash|sh|shell)\n(.*?)

```", re.DOTALL | re.IGNORECASE)
ENV = re.compile(r"\b([A-Z][A-Z0-9_]{2,})\b")
SKIP_ENV = {"HTTP", "GET", "POST", "JSON", "TODO", "README", "EOF"}

def extract_readme_claims(readme: Path) -> dict:
    text = readme.read_text(encoding="utf-8")
    commands: list[str] = []
    for block in FENCE.findall(text):
        for raw in block.splitlines():
            line = raw.strip()
            if not line or line.startswith("#"):
                continue
            commands.append(line)
    envs = sorted({m for m in ENV.findall(text) if m not in SKIP_ENV})
    return {"commands": commands, "env_names": envs, "readme": str(readme)}

Verify stage 1. Drop this next to the file and run it:

python - << 'PY'
from pathlib import Path
from extract import extract_readme_claims
claims = extract_readme_claims(Path("fixture/README.md"))
assert "npm run dev" in claims["commands"]
assert "DATABASE_URL" in claims["env_names"]
print("stage 1 ok", claims)
PY

If that assertion fails, your fence language tag is wrong. Fix the parser before you touch a model. Would you rather debug a regex now, or a rewritten README later?

Stage 2 — Inventory what the repo can actually run

A claim is only interesting when you can point at a file that should satisfy it. Inventory scripts, Make targets, and .env.example keys. Nothing fancy.

Give the fixture a different truth than the README. That is the point.

cat > fixture/package.json << 'EOF'
{
  "name": "demo",
  "scripts": {
    "start": "node server.js",
    "test": "pytest -q"
  }
}
EOF

cat > fixture/Makefile << 'EOF'
.PHONY: test
test:
    pytest -q
EOF

cat > fixture/.env.example << 'EOF'
DATABASE_URL=
PORT=8088
EOF

# inventory.py
from __future__ import annotations

import json
import re
from pathlib import Path

MAKE_TARGET = re.compile(r"^([a-zA-Z0-9][a-zA-Z0-9_-]*):", re.MULTILINE)

def inventory_repo(root: Path) -> dict:
    scripts: dict[str, str] = {}
    pkg = root / "package.json"
    if pkg.exists():
        data = json.loads(pkg.read_text(encoding="utf-8"))
        scripts.update((data.get("scripts") or {}))

    make_targets: list[str] = []
    makefile = root / "Makefile"
    if makefile.exists():
        make_targets = MAKE_TARGET.findall(makefile.read_text(encoding="utf-8"))

    env_example: list[str] = []
    env_file = root / ".env.example"
    if env_file.exists():
        for line in env_file.read_text(encoding="utf-8").splitlines():
            if not line or line.startswith("#") or "=" not in line:
                continue
            env_example.append(line.split("=", 1)[0].strip())

    return {
        "npm_scripts": scripts,
        "make_targets": make_targets,
        "env_example": env_example,
    }

Verify stage 2.

python - << 'PY'
from pathlib import Path
from inventory import inventory_repo
inv = inventory_repo(Path("fixture"))
assert "start" in inv["npm_scripts"]
assert "dev" not in inv["npm_scripts"]
assert "bootstrap" not in inv["make_targets"]
assert "REDIS_URL" not in inv["env_example"]
print("stage 2 ok", inv)
PY

See the gap? The README still sells npm run dev. The repo only knows start. That is drift. Not a style issue. A broken first run.

Stage 3 — Diff into a report a CI job can fail on

Classification rules stay boring. Boring is testable.

README claim	Looks like	Proven if
npm run X	npm script	X in package.json#scripts
make X	Make target	X in Makefile
pytest ...	tool on PATH	skip existence; mark unverified_tool
curl http://localhost:PORT/...	port	PORT in .env.example or the URL port matches
SOME_ENV	env var	key in .env.example

# diff_report.py
from __future__ import annotations

import json
import re
from pathlib import Path

from extract import extract_readme_claims
from inventory import inventory_repo

NPM_RUN = re.compile(r"^npm run ([a-zA-Z0-9:_-]+)")
MAKE = re.compile(r"^make ([a-zA-Z0-9_-]+)")
PORT = re.compile(r"localhost:(\d+)")

def classify(root: Path) -> dict:
    claims = extract_readme_claims(root / "README.md")
    inv = inventory_repo(root)
    findings = []

    for cmd in claims["commands"]:
        npm = NPM_RUN.match(cmd)
        mk = MAKE.match(cmd)
        port = PORT.search(cmd)
        if npm:
            name = npm.group(1)
            ok = name in inv["npm_scripts"]
            findings.append({"claim": cmd, "kind": "npm_script", "ok": ok, "expected": name})
        elif mk:
            name = mk.group(1)
            ok = name in inv["make_targets"]
            findings.append({"claim": cmd, "kind": "make_target", "ok": ok, "expected": name})
        elif port:
            expected = port.group(1)
            ok = expected in " ".join(inv["env_example"]) or any(
                expected in v for v in inv["env_example"]
            )
            # Port is proven only when .env.example mentions it explicitly.
            ok = any(expected in key or key == "PORT" and expected == _port_value(root)
                     for key in inv["env_example"])
            findings.append({"claim": cmd, "kind": "port", "ok": False, "expected": expected})
        else:
            findings.append({"claim": cmd, "kind": "unverified_tool", "ok": None, "expected": None})

    for env_name in claims["env_names"]:
        ok = env_name in inv["env_example"]
        findings.append({"claim": env_name, "kind": "env", "ok": ok, "expected": env_name})

    missing = [f for f in findings if f["ok"] is False]
    return {"inventory": inv, "findings": findings, "missing_count": len(missing)}

def _port_value(root: Path) -> str | None:
    env = root / ".env.example"
    if not env.exists():
        return None
    for line in env.read_text(encoding="utf-8").splitlines():
        if line.startswith("PORT="):
            return line.split("=", 1)[1].strip() or None
    return None

if __name__ == "__main__":
    report = classify(Path("fixture"))
    Path("drift_report.json").write_text(json.dumps(report, indent=2), encoding="utf-8")
    print("missing_count", report["missing_count"])

The port branch is intentionally strict. A curl to :3000 is false when .env.example says 8088. Do not let a model "fix" that by inventing a proxy.

Verify stage 3.

python diff_report.py
python - << 'PY'
import json
from pathlib import Path
report = json.loads(Path("drift_report.json").read_text())
missing = [f for f in report["findings"] if f["ok"] is False]
kinds = {f["kind"] for f in missing}
assert report["missing_count"] >= 3
assert "npm_script" in kinds
assert "make_target" in kinds
print("stage 3 ok; missing claims:")
for f in missing:
    print(" -", f["kind"], f["claim"])
PY

If missing_count is zero on this fixture, your classifier is too kind. Kind classifiers write pretty docs. They do not catch broken clones.

Stage 4 — Optional rewrite, only for proven-false claims

Here is the rule that keeps this from becoming another summary bot: the model never sees the full README. It only sees the ok: false rows plus the inventory. If a command is unverified_tool, you do not ask the model to bless it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode as a user of its free model access for this optional stage. Stages 1–3 do not need it.

Label the next function as an example. Point it at an OpenAI-compatible base URL you control. Do not hardcode secrets. Do not invent a model name in CI.

# rewrite.py — labeled example, not a production client
from __future__ import annotations

import json
import os
import urllib.request
from typing import Any

SYSTEM = """You rewrite getting-started commands.
You receive JSON with inventory and missing claims.
Return JSON only: {"replacements": [{"claim": str, "suggested": str, "reason": str}]}.
Suggested commands MUST use names that already exist in inventory.
If you cannot map a claim, set suggested to null.
Do not invent scripts, ports, or env vars.
"""

def rewrite_missing(report: dict[str, Any]) -> dict[str, Any]:
    base = os.environ.get("MODEL_BASE_URL", "").rstrip("/")
    token = os.environ.get("MODEL_API_TOKEN", "")
    if not base or not token:
        raise SystemExit("skip rewrite: set MODEL_BASE_URL and MODEL_API_TOKEN")

    missing = [f for f in report["findings"] if f["ok"] is False]
    payload = {
        "model": os.environ.get("MODEL_NAME", ""),
        "messages": [
            {"role": "system", "content": SYSTEM},
            {
                "role": "user",
                "content": json.dumps(
                    {"inventory": report["inventory"], "missing": missing},
                    indent=2,
                ),
            },
        ],
        "temperature": 0,
    }
    req = urllib.request.Request(
        base + "/chat/completions",
        data=json.dumps(payload).encode("utf-8"),
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + token,
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        body = json.loads(resp.read().decode("utf-8"))
    content = body["choices"][0]["message"]["content"]
    return json.loads(content)

Verify stage 4 without calling anything. Gate the model with a local contract test. If the model violates it, you drop the output.

# test_rewrite_contract.py
from inventory import inventory_repo
from pathlib import Path

def assert_replacements_are_grounded(inventory: dict, replacements: list[dict]) -> None:
    npm = set(inventory["npm_scripts"])
    makes = set(inventory["make_targets"])
    envs = set(inventory["env_example"])
    for row in replacements:
        suggested = row.get("suggested")
        if suggested is None:
            continue
        if suggested.startswith("npm run "):
            name = suggested.split()[-1]
            assert name in npm, suggested
        elif suggested.startswith("make "):
            name = suggested.split()[-1]
            assert name in makes, suggested
        elif suggested.isupper():
            assert suggested in envs, suggested

def test_fixture_mapping():
    inv = inventory_repo(Path("fixture"))
    # Simulated model output — labeled, not executed against a live API.
    fake = [
        {"claim": "npm run dev", "suggested": "npm run start", "reason": "script exists"},
        {"claim": "make bootstrap", "suggested": None, "reason": "no equivalent target"},
    ]
    assert_replacements_are_grounded(inv, fake)

python -m pytest test_rewrite_contract.py -q
echo "stage 4 contract ok"

That test is the real product. The HTTP call is optional. If a free model maps npm run dev to npm run start, good. If it invents npm run bootstrap-all, the contract kills it. Why would you merge a sentence you cannot grep?

When I did wire this optional stage, MonkeyCode's free model access was enough to return JSON replacements. The free server option is useful only if you want the audit off your laptop. Neither one replaces the diff.

Stage 5 — Run it like a gate, not like a chatbot

Wrap the classifier so CI can fail on missing_count.

# verify.sh
set -euo pipefail
python diff_report.py
python - << 'PY'
import json, sys
from pathlib import Path
report = json.loads(Path("drift_report.json").read_text())
print("missing_count", report["missing_count"])
sys.exit(1 if report["missing_count"] else 0)
PY

Verify stage 5. On the fixture this must exit 1.

bash verify.sh; echo "exit=$?"

You want a red build. A green build on this fixture means you are measuring the wrong thing.

If you later schedule verify.sh on a small free server, keep the same contract: no rewrite in the failing job. Generate suggestions in a second, non-gating step. Mix those and you cannot tell a parser bug from a model bug.

Limitations, said plainly

This does not execute commands. pytest -q can be listed and still crash. Fences that omit a language tag are invisible. docker compose services, heredocs, and Makefile variables will fool the inventory. Env names harvested from prose will false-positive on HTTP unless you keep the skip list honest.

It also will not fix architecture. A README can be fully consistent and still describe the wrong system. That is a human problem.

Who should not use this

Skip it if your getting-started path is generated from the same source as your scripts. Skip it if the README is a marketing page with no fences. Skip it if you need a threat model, a license audit, or a promise that curl succeeds. And skip the rewrite stage if you cannot fail a build on ungrounded suggestions.

The conclusion does not change if you never call a model. Prove the commands. Then, if you want a draft of the replacements, use free model access as a bounded JSON rewriter — I did that with MonkeyCode, once the report already existed. The report is the work.