I let an AI agent write a README for a real repo — here's what it produced

Most small open-source projects have a README that stopped matching the code
two years ago. I wanted to know whether an AI agent could write one that
actually reflects what's in the repository — not a plausible-sounding one, but
one where every flag and environment variable it mentions is real.

So I built a small pipeline and pointed it at someone else's repository.

The setup

The rule I started with: the model never decides what's true about the code.

The pipeline has four stages, and only one of them involves an LLM:

1. Read — fetch the repository's files over HTTP.
2. Extract facts — parse pyproject.toml, argparse calls, os.environ lookups, the LICENSE file. Deterministic, no model involved.
3. Generate — hand the model the files and the extracted facts, ask for a README.
4. Verify — check the output against the facts. Any CLI flag, environment variable, or license claim that isn't in the extracted facts is treated as invented, and the deliverable is rejected.

Stage 4 is the part that makes this usable. A README that documents a
--dry-run flag which doesn't exist is worse than no README at all, because
someone will try it.

The repository

sloria/ped — a small MIT-licensed CLI tool
that opens Python modules in your editor. A few hundred lines across a
handful of source files. Small enough to reason about, real enough to be interesting.

I picked it because it already has a decent README. That made it a fair
test: could the agent produce something comparable, without copying it?

What the agent read

Nine files, chosen by a priority list — config first, then entry points, then
the rest:

LICENSE
README.rst
pyproject.toml
src/ped/__init__.py
src/ped/guess_module.py
src/ped/install_completion.py
src/ped/ped_bash_completion.sh
src/ped/pypath.py
src/ped/style.py

What it extracted, deterministically

{
  "name": "ped",
  "version": "3.0.0",
  "license": "MIT",
  "requires_python": ">=3.8",
  "scripts": ["ped"],
  "cli_flags": ["--complete", "--editor", "--help", "--info",
                "--version", "-e", "-h", "-i", "-v"],
  "env_vars": ["EDITOR", "NO_COLOR", "PED_EDITOR",
               "PED_OPEN_DIRECTORIES", "SHELL", "VISUAL"]
}

Two of these were harder to get than they look.

--help and -h are never written in the source — argparse adds them. If
you extract flags naively and then verify against that list, a correct README
that documents --help gets rejected as hallucinated. I had to add them
explicitly as known-implicit.

The environment variables were worse. The code reads them in a loop:

for key in "PED_EDITOR", "VISUAL", "EDITOR":
    rv = os.environ.get(key)

A regex looking for os.environ.get("SOMETHING") finds nothing here. My first
verifier flagged PED_EDITOR as invented — in a README that was correct. The
extractor was wrong, not the text.

That failure was useful. It's the exact shape of error that makes verification
worth building: the checker caught a mismatch, and the mismatch turned out to
be in the checker.

What it produced

The full output is
here. A few things it
got right that I didn't expect:

The environment variable precedence. It documented that PED_EDITOR takes
priority over VISUAL, which takes priority over EDITOR — as a table. That
ordering is only visible from the loop above; nothing states it in prose.

PED_OPEN_DIRECTORIES. An undocumented-in-README variable that changes
whether a package opens as a directory or as its __init__.py. It's one line
in the source. The agent found it, explained what it does, and gave a shell
snippet for setting it.

Tab completion. It read install_completion.py, worked out that the module
prints a completion script to stdout and picks bash vs zsh from $SHELL, and
wrote a section for it.

How it works. Four steps, each traceable to actual code — sys.path
manipulation for pipx installs, difflib for partial name matching, inspect
for locating source and line numbers, and the specific list of editors that
get a +lineno argument.

What went wrong

One thing, and it's instructive.

The Bash tab-completion section came out as:

python -m ped.install_completion > /usr/local/etc/bash_completion.d

That redirects into a directory, which fails. My first instinct was that the
agent had invented it.

It hadn't. That exact command is in the upstream README. The agent reproduced
the source faithfully — including its mistake.

This is a real limit of the approach, and I'd rather state it than hide it:
the pipeline verifies that claims match the code, not that the code's own
documentation is correct. When a project's README has an error, a
code-grounded rewrite can carry it forward.

I changed it to a generic form that can't be pasted into an invalid path:

# Redirect the output to a file inside your shell's completion directory.
# The exact location depends on your OS.
python -m ped.install_completion > <bash-completion-dir>/ped

I didn't invent /ped as the filename, because nothing in the code specifies
a destination — install_completion.py just writes to stdout. Where it goes
is the user's choice.

The verification output

cli_flags   mentioned=8   invented=[]
env_vars    mentioned=5   invented=[]
sections    12
trailing newline: ok
requires_python stated: yes

Zero invented flags, zero invented environment variables. Not because the
model is trustworthy — because anything it invented would have been caught and
the whole deliverable rejected.

Cost

$0.04 for the generation. One call, ~3,500 output tokens.

The reading and verification added no LLM cost; they run as deterministic
code. That ratio matters: the expensive part is the writing, and the writing is the
part you can't trust without checking.

What I'd take away from this

Extraction is harder than generation. The model wrote a good README on the
first try. Getting a reliable list of what's actually in the code took
several iterations, and in this run the bugs I found were in my parser
rather than in the generated prose.

Verification changes what you can ship. Without stage 4, this is a demo. A
README that's 95% right is unusable for anything real, because you don't know
which 5%. With a checker that rejects unverifiable claims, the failure mode
becomes "output rejected" instead of "output subtly wrong."

Faithful isn't the same as correct. The completion-path bug is the clearest
lesson here. Grounding output in source code means inheriting the source's
errors too. A human still has to read the result.

---

I turned this into a small service: send a public GitHub repository URL, get
a README back within 24 hours, £30, and you see the full file before you pay
anything. It's run by an AI agent that I supervise — I read every deliverable
before it's sent. Details at
agent.aitomaton.dev.

The example above is the unedited pipeline output, apart from the one
correction described. Compare it with
the repository's own README if you want to
judge for yourself.