TL;DR — I built a tool that authors AGENTS.md from real repo facts, never invented ones. A month after it was scored as a real AAIF contribution: 29 downloads, 2 stars, 0 issues. Nobody stress-tested it, so I did — against two repos I'd already dug into by hand. Every line it wrote was true. It still missed the most important test in both of them, for the same structural reason, twice.
The honest starting number
agents-md-facts authors a minimal AGENTS.md from what your repo actually declares — real build commands, real test commands, real file layout. Nothing guessed. It was submitted to AAIF as a project contribution and scored accordingly.
A month later: 29 npm downloads, 2 GitHub stars, 0 issues, 0 forks. Five commits since launch, all docs and housekeeping — nobody used it hard enough to find something to fix.
That's not a failure post. It's the honest premise for this one: if nobody else was going to stress-test it, I would.
Dogfooding it for real
I ran it, --dry-run, against two repos I already understood deeply — not blind spots, ground truth I could check the output against. One Rust, one TypeScript.
$ cd mcp-better && npx agents-md-facts --dry-run
## Run the tests
```bash
cargo test
cargo clippy
```
## Definition of Done
Done when: `cargo clippy` exits 0 · `cargo test` passes · committed with a clear message.
$ cd claude-faf-mcp && npx agents-md-facts --dry-run
## Run the tests
```bash
npm run test
npm run lint
```
## Definition of Done
Done when: `npm run lint` exits 0 · `npm run test` passes · committed with a clear message.
What it got right
Every single line in both outputs traces to something real. cargo test and cargo clippy genuinely exist and genuinely run in mcp-better. npm run test and npm run lint genuinely exist and genuinely run in claude-faf-mcp. Zero invented commands, either time. The tool's actual promise — never guessed — held completely, both times, under real conditions.
What it missed — twice, same shape
mcp-better ships a tool called confirm_echo with a real multi-round contract: a sealed, tamper-checked handshake. The unit tests for it live in src/server.rs and run fine under cargo test. But the contract — does the tool still promise what it promised, over the wire, to a real client — is checked by a completely separate command:
$ cargo test --all-targets
Running unittests examples/mrtr_client.rs
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored
cargo test compiles that file. It runs zero test functions in it. The actual check only happens if you separately run cargo run --example mrtr-client — which is exactly the command the generated AGENTS.md never mentions, because "Definition of Done: cargo test passes" is what the tool detected, and that line is true. It's just not the whole truth.
claude-faf-mcp has the same shape from a different angle. The generated file correctly points at npm run test. Sitting in the same tests/ directory, untouched by the generated output: WJTTC-FAFM-MEMORY.md, WJTTC-MCP-CLI-CONTINUITY-v274.md, WJTTC-REPORT-MCP-v120-STRESS-TEST.md — a separate certification layer the tool has no way to surface, for the same reason: it doesn't look like npm run test, so there's nothing for a facts-detector to grab onto.
Two different stacks. Same blind spot, in the same place: the deepest verification layer in each repo is precisely the one that doesn't look like every other repo's test command.
Why — and it isn't a bug
"Never guessed" means the tool only writes down things it can detect. That's the entire safety property, and it's a real one — it's the whole reason this class of tool doesn't fall into the auto-generated-bloat trap the research on AI-written instruction files warns about. But detection has a shape. The tool knows what cargo test and npm run test look like. It has no way to know that cargo run --example mrtr-client or a hand-written stress report also answers "is this thing actually verified" — because those don't match any pattern it's built to recognize.
"Real fact" and "fact my detector recognizes" are not the same set. The gap between them is exactly where the most important line in an AGENTS.md can go missing, silently, while every other line stays true.
What this means beyond this one tool
Any facts-based generator — this one or the next one — inherits the same limit. A generated AGENTS.md is a floor, not a ceiling: provably not-wrong, because every line traces to something real, which is not the same claim as complete. And the layer most likely to be missing is usually the one that matters most, because a repo's deepest verification is often exactly the thing that was built custom, in a shape nothing else in that codebase uses — which is precisely what a pattern-matcher is worst at finding.
What I'm changing because of this
The fact-sourcing discipline underneath all of this — every line traces to a fact, verify it stays true in the same PR it changed — is still the right foundation. It just isn't sufficient alone, and pretending otherwise would be exactly the kind of overclaim this series has spent five parts arguing against.
The honest addition is one more pass, after the tool runs, human or agent: does this repo have a second, differently-shaped verification layer the tool wouldn't have found? That's the question that surfaced mrtr-client and the WJTTC reports here. It's not automatable yet. Naming it is the first step toward making it so.
Help guide what we build — comments and suggestions welcome.
Top comments (1)
That distinction is exactly where evaluation gets slippery. A system can avoid guessing and still fail by staying silent at the wrong time. I like measuring abstention separately: when did it refuse correctly, when did it miss a recoverable answer, and what evidence would have changed the decision?