Weekend Build Log: A Docs Drift Checker That Runs Your README

Last Saturday a teammate pasted our README quickstart into a fresh container. Step three failed. The install command had been renamed four months ago.

Nobody noticed, because nobody runs docs. We run tests.

So I gave myself one weekend. Build the smallest tool that executes every fenced shell block in our docs. Fail loudly when a block breaks. Ship a working demo by Sunday night.

This is the build log. It covers the scope cuts, the code, and the parts I deliberately skipped.

Why a checker instead of a linter

Linters read docs. A checker runs them.

That difference matters more than it sounds. A linter says a command looks odd. A checker says the command exits 3 on a clean machine.

I wanted the second kind. So the first design rule was strict.

Rule zero: the model never decides pass or fail

Deterministic code decides whether a block drifted. A language model never touches that verdict.

I learned this on an earlier side project. A model that classifies failures is fun to demo and painful to trust. It drifts, it costs money per run, and it invents confident reasons.

So the split stays clean:

1. Python extracts the shell blocks.
2. Docker runs each block and returns an exit code.
3. The exit code decides pass or fail.
4. A model gets one bounded job: turn raw stderr into one readable line.

If the model is unavailable, the checker still works. That property is worth more than any prompt.

Step 1: Extract the fenced blocks

Start with the smallest parser that could work. No Markdown library, no AST, no plugin system.

#!/usr/bin/env python3
'''driftcheck.py - execute fenced shell blocks from Markdown files.'''
from __future__ import annotations

import argparse
import json
import re
import subprocess
import sys
import time
from pathlib import Path

FENCE = re.compile(r'```

(?:bash|sh|console)\n(.*?)

```', re.DOTALL)
PROMPT = re.compile(r'^\s*[$#]\s?')


def extract(path: Path) -> list[tuple[int, str]]:
    '''Return (block_index, script) for every fenced shell block.'''
    blocks = []
    for i, raw in enumerate(FENCE.findall(path.read_text()), start=1):
        lines = [PROMPT.sub('', line) for line in raw.splitlines()]
        script = '\n'.join(line for line in lines if line.strip())
        if script:
            blocks.append((i, script))
    return blocks

Two details earn their keep here. The PROMPT regex strips the $ you copy from terminal output. The empty-line filter stops trailing blanks from confusing sh -c.

Step 2: Run each block in a throwaway container

Never run docs commands on your own machine. You do not know what a contributor pasted in.

def run_block(script: str, image: str, workdir: Path, timeout: int) -> dict:
    started = time.monotonic()
    proc = subprocess.run(
        [
            'docker', 'run', '--rm',
            '--network=none',
            '-v', f'{workdir}:/work',
            '-w', '/work',
            '--entrypoint', 'sh',
            image, '-c', script,
        ],
        capture_output=True,
        text=True,
        timeout=timeout,
    )
    return {
        'exit_code': proc.returncode,
        'seconds': round(time.monotonic() - started, 2),
        'stderr_tail': proc.stderr[-400:],
    }

--network=none carries the weight here. Blocks that secretly download something now fail fast and loudly.

Step 3: Report, then set the exit code

def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument('docs', nargs='+', type=Path)
    ap.add_argument('--image', default='python:3.12-alpine')
    ap.add_argument('--timeout', type=int, default=60)
    ap.add_argument('--json-out', type=Path)
    args = ap.parse_args()

    report, drifted = [], 0
    for doc in args.docs:
        for index, script in extract(doc):
            result = run_block(script, args.image, doc.parent.resolve(), args.timeout)
            result.update({'doc': str(doc), 'block': index})
            report.append(result)
            if result['exit_code'] != 0:
                drifted += 1
                print(f'DRIFT {doc} block {index} exit={result[chr(34)+chr(34)]}')

    if args.json_out:
        args.json_out.write_text(json.dumps(report, indent=2))

    print(f'blocks={len(report)} drifted={drifted}')
    return 1 if drifted else 0


if __name__ == '__main__':
    sys.exit(main())

Ignore that f-string edit and write it plainly. Use exit={result['exit_code']} in your own copy.

The exit code is the product. Everything else is decoration. Wire this into CI and a broken block becomes a red build.

Step 4: Add one bounded model call

Now the model earns its place. It reads stderr and names the failing command. Nothing else.

import httpx

# Provider-agnostic. Swap base_url and model for whatever your free tier exposes.
def hint(stderr: str, base_url: str, model: str, key: str) -> str:
    payload = {
        'model': model,
        'max_tokens': 60,
        'messages': [
            {'role': 'system',
             'content': 'Name the failing command in one line. No advice.'},
            {'role': 'user', 'content': stderr[-1500:]},
        ],
    }
    resp = httpx.post(
        f'{base_url}/chat/completions',
        json=payload,
        headers={'Authorization': f'Bearer {key}'},
        timeout=20,
    )
    resp.raise_for_status()
    return resp.json()['choices'][0]['message']['content'].strip()

Three limits keep this honest. Cap max_tokens at 60. Cap the request at 20 seconds. Drop the call on failure and keep the raw stderr.

Redact before you send. Stderr from a failed login block can contain tokens.

Step 5: Keep execution local, move the always-on part

Here is where MonkeyCode changed my weekend plan. My laptop cannot be the webhook receiver.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option, and I used both in this build. The free server hosts the webhook and the report endpoint. It is always on, which is the only property I needed. The free model endpoint handles the single hint call per drifted block, and nothing more. Check the current free-tier terms on their site before you design around them, since allowances and limits change.

One thing I refused to move: the execution. Running untrusted shell on shared free infrastructure is how you get a very bad Monday. The container run stays on a machine I control.

curl -sS -X POST https://<your-free-server>/hook \
  -H 'content-type: application/json' \
  -d '{"repo":"docs","reason":"push"}'

The receiver stores the request and pings me. My local runner picks it up, runs the blocks, and POSTs the JSON report back. Two moving parts, no shared trust.

The acceptance test I ran before shipping

Do not trust a checker you have not seen fail. Two fixtures prove the contract.

mkdir -p tests/fixtures
printf '```

bash\necho ok\n

```\n' > tests/fixtures/good.md
printf '```

bash\nexit 3\n

```\n' > tests/fixtures/bad.md

python driftcheck.py tests/fixtures/good.md; echo "exit=$?"   # exit=0
python driftcheck.py tests/fixtures/bad.md;  echo "exit=$?"   # exit=1

python driftcheck.py tests/fixtures/bad.md --json-out /tmp/r.json
python -c "import json; print(json.load(open('/tmp/r.json'))[0]['exit_code'])"  # 3

If the good fixture does not return 0, your parser is broken. If the bad fixture does not return 1, your CI gate is broken. Run both before you wire anything to a pipeline.

The scope cuts I made

A weekend project dies from ambition, not from bugs. Here is the cut list.

Shipped	Skipped	Why
bash, sh, console blocks	Python, JS, TS blocks	Each language needs its own image and install step
Serial execution	Parallel runs	A dozen blocks finish fast enough in sequence
Exit code plus JSON report	HTML dashboard	The log line is the deliverable
One-line model hint	Auto-fix pull requests	Auto-edits to docs create review work, not savings
Local runner, remote webhook	Remote execution	Untrusted shell does not belong on shared infra

Two of those cuts hurt. Skipping Python blocks left real docs uncovered. Skipping parallel runs made the first CI version slow. Both were still the right call for one weekend.

Limitations and who should not use this

Be honest about the edges before you copy the approach.

• Do not run blocks that need real cloud credentials. Put them behind a manual checklist instead.
• Docs that intentionally print broken output will always report drift. Exclude those files.
• Teams without a container runtime cannot isolate anything. Use a disposable VM instead.
• If you want automatic fixes, this tool will disappoint you. It reports, it does not repair.
• Every hint call sends stderr to a third party. Redact secrets first, or run the model locally.

What I would build next

The next step is a config file listing blocks to never run. After that, a per-language image matrix. Both can wait.

The demo runs. The model is optional. The exit code is the product.

If you want to try the same split, MonkeyCode's free model access and free server option are enough to keep a weekend bill at zero. Start with the exit code, not the prompt.

Comments

[Launch Gate]

The bind mount is the boundary I'd tighten before running contributor-supplied blocks. As shown, -v ...:/work is writable, so a container can still change or delete files in the checked-out docs directory even with networking disabled. Could execution use a disposable copy, with the original mounted read-only only where needed? A sentinel-file fixture would make the guarantee visible: let a block try to overwrite it, then verify the original is unchanged. I'd also test false; echo ok, since the shell's final exit status can hide an earlier failed command.