DEV Community

Cover image for The AI that refuses to publish a tutorial it didn't run twice
bishwas jha
bishwas jha

Posted on • Originally published at alphacrack.github.io

The AI that refuses to publish a tutorial it didn't run twice

#opensource #ai #python #devtools

There's a special kind of betrayal in a README quickstart that doesn't work.

You paste the command. It fails. You backtrack, was it a missing step, a flag that got renamed three releases ago, a dependency the author had installed so long ago they forgot to mention it? The instructions were written by a human who was sure they worked. They just… didn't, not on a clean machine.

AI has made this worse, not better. Point a language model at a repo and it will happily generate a confident, well-formatted tutorial — one that was never executed a single time. It reads plausible. That's the problem. Plausible is not the same as runs.

I built readme2demo to take the opposite stance: it doesn't trust the model, and it doesn't trust itself. It trusts a container.

The idea in one sentence

Point it at a repo. An AI agent reads the README and actually runs it inside a hardened Docker sandbox. The working path is distilled to its minimum, then replayed in a brand-new container the agent never touched — and only what survives that clean-room replay gets published.

Out the other end you get a tutorial.md, a step-by-step guide, a
troubleshooting doc, a machine-readable howto.jsonld, and a VHS demo video that types every verified command on camera.

The value was never "AI writes a tutorial." It's that the tutorial ran, twice, before you saw it.

The one rule everything is built around

The LLM never publishes anything a fresh container did not independently execute.

This is the whole project in a sentence, and the important part is how it's enforced: in code, not in prompts.

Prompts are suggestions. A model told "only include commands that worked" will, eventually, helpfully include one that didn't. So grounding isn't left to the model's good intentions. Every command the distiller wants to emit has to fuzzy-match a command that actually succeeded in the recorded run log. No match, no publish. And the final verdict doesn't come from the agent at all — it comes from a separate stage that replays the distilled script in a container with zero agent state.

If that replay fails, the output doesn't get quietly patched. It ships loudly labeled ⚠ UNVERIFIED. When it passes, every doc carries a badge:

✅ Verified on 2026-07-07 · image <digest> · commit <sha>
Enter fullscreen mode Exit fullscreen mode

How it actually works

It's a pipeline of small, resumable stages over a crash-safe manifest.json, so a run that dies at stage 5 resumes at stage 5 instead of starting over:

repo URL / guide → ingest & plan → agent run (in Docker) → normalize transcript
        → distill minimal path → VERIFY replay in a fresh container
        → tutorial + step_by_step.md → render VHS demo video
Enter fullscreen mode Exit fullscreen mode
  1. ingest — clone the repo, collect the README/docs, and run one planner pass that emits a machine-checkable plan with a feasibility verdict. Quickstarts that can't work in a sandbox (need a GPU, real cloud creds, a GUI) fail here, for pennies, before any agent time is spent.
  2. agent — the agent engine runs inside the hardened sandbox until the quickstart works, blocks, or runs out of turns.
  3. normalize — pure, deterministic Python turns the messy transcript into a structured command log. No LLM calls, fully testable against fixtures.
  4. distill — one LLM pass reduces the run to the minimal reproduction path and writes commands.sh. The grounding validator lives here.
  5. verify — the moat. commands.sh is replayed in a fresh container the agent never saw. This is the only source of the word "verified."
  6. tutorial — finalizes the step-by-step guide with the verified outputs.
  7. render — builds the VHS video from the finalized guide, so the demo provably follows the published steps line for line.

Because READMEs are untrusted code, the agent's container is the permission
boundary: cap-drop ALL, no-new-privileges, non-root execution, and
memory/CPU/PID limits.

The unglamorous part: everything that goes wrong

The honest engineering story here isn't the happy path — it's the long tail of
ways "run a README" breaks. A few that shaped the design:

  • The agent fakes success. Told to make something work, a model will patch the source, stub a socket, or pipe a failing command into head so it exits
    1. Each of these is now a detected failure class with a code-level defense and a regression test, not just a stern prompt.
  • Syntax drift breaks grounding. 2>&1, an env-var prefix, a | head, a heredoc body — the "same" command shows up a dozen ways. Matching is normalized on both sides so real successes don't get dropped as unverified.
  • Findings tools exit nonzero on success. Linters and drift-detectors "fail" when they find something. Under set -e that aborts the script before the assertion, so those are special-cased to still count as passing.
  • Videos that lie. The demo tape is generated from the finalized guide in the render stage — not hand-written — so the video can't drift from the doc.

Every one of those was found on a real run and has a test named after it. That's
the culture of the project: when you need the model to behave, you add both a
prompt rule and a parser that enforces it. Prompts are suggestions; parsers are
law.

Running it 

pip install -e ".[dev]"
docker build -t readme2demo/base:latest images/base/

# self-hosted, single-operator, on your Claude subscription (no API key):
claude setup-token
export CLAUDE_CODE_OAUTH_TOKEN=sk-ant-oat01-...
readme2demo run https://github.com/owner/repo --llm-backend claude-cli

# or metered API billing (best for scale, required if you host it for others):
export ANTHROPIC_API_KEY=sk-ant-...
readme2demo run https://github.com/owner/repo
Enter fullscreen mode Exit fullscreen mode

The repo is optional now — hand it a self-contained step-by-step guide instead
and the fresh-container replay still verifies every command.

What it is and isn't

It's open-core and MIT-licensed. The CLI and the whole verification pipeline
are the free core. A hosted version is exploratory — gated entirely on whether
people actually want it — and I'm deliberately not putting SaaS code into the
OSS core until that's answered.

It's honest about its edges: the in-sandbox agent needs a model credential that
lives in the container for the duration of the run (hardened, but a real
tradeoff), the OpenHands engine is experimental, and "verified" means the
quickstart runs — not that the project is good.

It even generated its own tutorial and demo. That self-run is committed in the
repo, alongside a run against a second project, so you can inspect exactly what
it produces before installing anything.

Repo: https://github.com/alphacrack/readme2demo (MIT)
Docs: https://alphacrack.github.io/readme2demo/

If the idea of docs that can't lie about whether they run appeals to you, a star
helps it find the next person and I'd genuinely like to hear where it breaks on
your repo.

Top comments (0)