The Changelog Had a Flag. The Parser Did Not.

Release notes can lie without a single test turning red.
I learned that the hard way while reviewing a CLI patch that looked polite.
The diff added --json-pretty to CHANGELOG.md. argparse never grew that option.
Would your pipeline catch that, or would the docs ship first?

This is not a model bake-off. It is a reviewer workflow for a boring, expensive class of drift: an agent (or a tired human) documents a flag that the parser cannot accept. The suite stays green because nothing executes the markdown.

I now treat changelog bullets as claims against code. If the claim has no add_argument, the notes do not merge. Full stop.

Why markdown is a lousy contract

Changelogs are prose. Parsers are not.
A sentence like "added pretty JSON output" can mean a new flag, a default change, or a formatter tweak. Agents love the first reading. They invent --json-pretty because it sounds like an API.

Do you assert release notes anywhere today?
Most repos do not. CI lints Python. CI may even snapshot --help. It rarely diffs --help against the newest changelog section. That gap is where ghost flags live.

Ghost flags hurt users in a specific way. They copy a bullet into a terminal, get unrecognized arguments, and blame themselves. Support then spends a day proving the docs were fiction.

Two files I actually trust

I only trust two artifacts for this check.

1. The module that calls parser.add_argument.
2. The newest version section in CHANGELOG.md.

README marketing copy stays out of the contract. Blog posts stay out. Chat transcripts stay out. If a flag is real, argparse knows. If a release claims a user-facing switch, the changelog must name it the same way the parser does, including the long option.

Short options are optional in the notes. Long options are not. Why? Because -j collides across tools and agents hallucinate letter codes. --json is the stable handle.

A tiny extractor, not a platform

I keep the checker in-repo on purpose. No service. No dashboard. Three files are enough.

flag_contract/
  cli.py
  extract.py
  test_changelog_flags.py

cli.py is a stand-in for your real entry point. Replace it with the module you actually ship. The next block is a teaching fixture, not a product CLI.

# cli.py
import argparse

def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(prog="invoice")
    parser.add_argument("--json", action="store_true", help="print JSON")
    parser.add_argument("--out", required=True, help="output path")
    parser.add_argument("-q", "--quiet", action="store_true")
    return parser

def main(argv=None) -> int:
    args = build_parser().parse_args(argv)
    if args.quiet:
        return 0
    print({"out": args.out, "json": args.json})
    return 0

Notice what is missing. There is no --json-pretty. That absence is the whole point of the fixture.

Walk argparse without running --help

Why not shell out to --help?
Because help text wrapping splits long options, and agents rewrite help epilogs. AST walking stays on the add_argument calls. It is narrower. Narrow is good here.

# extract.py
from __future__ import annotations

import ast
import re
from pathlib import Path

LONG_FLAG = re.compile(r"--[a-zA-Z0-9][a-zA-Z0-9-]+")
SECTION = re.compile(r"^## \\[[^\\]]+\\]", re.M)

def argparse_long_flags(source: str) -> set[str]:
    tree = ast.parse(source)
    found: set[str] = set()
    for node in ast.walk(tree):
        if not isinstance(node, ast.Call):
            continue
        func = node.func
        name = getattr(func, "attr", None) or getattr(func, "id", None)
        if name != "add_argument":
            continue
        for arg in node.args:
            if isinstance(arg, ast.Constant) and isinstance(arg.value, str):
                if arg.value.startswith("--"):
                    found.add(arg.value)
    return found

def newest_changelog_section(text: str) -> str:
    hits = list(SECTION.finditer(text))
    if not hits:
        raise ValueError("no version heading like ## [0.4.0]")
    start = hits[0].end()
    end = hits[1].start() if len(hits) > 1 else len(text)
    return text[start:end]

def changelog_long_flags(section: str) -> set[str]:
    return set(LONG_FLAG.findall(section))

def load(path: str) -> str:
    return Path(path).read_text(encoding="utf-8")

The regex on markdown is deliberately greedy. It will also pick up example commands. That is a feature. If the notes demonstrate --json-pretty, the parser must accept --json-pretty.

A changelog that fails on purpose

Label this file as a failing fixture. Do not copy it into a real release.

## [0.4.0] - 2026-09-22

### Added
- `--json-pretty` pretty-prints API payloads
- `--out` is now required so CI cannot dump to stdout by accident

### Fixed
- `--quiet` no longer prints the debug banner

Three long flags appear in that section. Only two exist in cli.py. Which one is the ghost? --json-pretty. The required check is not "did the model sound confident." The required check is set difference.

Diff them like a reviewer

# test_changelog_flags.py
from extract import argparse_long_flags, changelog_long_flags, load, newest_changelog_section

def test_newest_notes_do_not_invent_flags():
    parser_flags = argparse_long_flags(load("cli.py"))
    section = newest_changelog_section(load("CHANGELOG.md"))
    note_flags = changelog_long_flags(section)
    ghosts = sorted(note_flags - parser_flags)
    assert ghosts == [], f"changelog invented flags: {ghosts}"

def test_newest_notes_mention_new_required_flags():
    # Optional tighter rule: required flags introduced in this module
    # must appear in the newest section. Relax this if you version slowly.
    parser_flags = argparse_long_flags(load("cli.py"))
    section = newest_changelog_section(load("CHANGELOG.md"))
    note_flags = changelog_long_flags(section)
    missing_docs = sorted({"--out"} - note_flags)
    assert missing_docs == [], f"required flags absent from notes: {missing_docs}"

Run it locally.

python -m pytest test_changelog_flags.py -q

Expected failure, if the fixture is intact:

AssertionError: changelog invented flags: ['--json-pretty']

That line is the review comment I want. Not "please improve the wording." Not "LGTM, the model is usually right."

A table I actually fill

I fill this from the two sets. I do not fill it from memory, and I do not fill it from a chat window.

Long flag	In argparse	In newest notes	Reviewer action
--json	yes	no	ignore, or document later
--out	yes	yes	keep
--quiet	yes	yes	keep
--json-pretty	no	yes	delete the bullet or add the argument

Four outcomes matter. I keep them boring on purpose.

1. In parser, not in notes: allowed, unless the flag is new and required.
2. In both: merge candidate.
3. In notes, not in parser: block. This is the ghost.
4. In neither, but mentioned in a PR description: ignore. PR text is not a contract.

If row 3 is non-empty, the changelog is the bug. Do not "fix" it by weakening the test. Do not rename the test to test_changelog_style. Style is not the failure.

Commands I run before I merge docs

The sequence is short. I refuse extra ceremony.

1. Freeze the newest heading. If the agent rewrote history under an old version, stop. Changelogs are append-only in this workflow.
2. Print both sets.
3. Fail on ghosts.
4. Only then read the prose for tone.

python - <<'PY'
from extract import argparse_long_flags, changelog_long_flags, load, newest_changelog_section
parser_flags = argparse_long_flags(load("cli.py"))
notes = changelog_long_flags(newest_changelog_section(load("CHANGELOG.md")))
print("parser", sorted(parser_flags))
print("notes ", sorted(notes))
print("ghosts", sorted(notes - parser_flags))
print("undoc ", sorted(parser_flags - notes))
PY

Undocumented parser flags are a judgment call. Ghosts are not. Why the asymmetry? Missing docs annoy experts. Invented docs break newcomers.

Where a free coding box actually helps

I still draft the notes with a model. I do not merge the draft.
The useful split is: model proposes bullets, extractor vetoes ghosts, human writes the one-line user impact.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open source coding project with free model access and a free server option. The operator-supplied offering I am using here is that free model access (including a free 10 million token allotment advertised for the project) plus a short-lived free server so the extractor can run outside my laptop. I use the model to rewrite a changelog section. I use the server to run pytest on a clean checkout. I do not use either as evidence. Evidence is the set difference printed above.

If the server is busy, the same commands run locally. The workflow does not depend on a vendor remaining free forever. Treat availability as a convenience, not a promise.

What this check refuses to claim

It does not prove the flag behaves. store_true versus store is out of scope.
It does not parse Click, Typer, Cobra, or clap. Those need their own walkers.
It does not read RST, AsciiDoc, or HTML release pages.
It does not understand "deprecated --out in favor of --output." Aliases need an explicit map, and I do not hide that map in prose.

Hand-written add_argument inside a loop will evade the AST walk. So will flags registered from a JSON schema at runtime. If your CLI is a plugin host, this checker is too small. Say that in the PR, then write a runtime dump of parser._option_string_actions instead.

Dates in headings are not validated. I do not care if the agent stamped tomorrow. I care if it stamped a switch that cannot parse.

Skip this if your CLI is generated

Skip the markdown regex if a generator already emits both the parser and the notes from one schema. Then the schema is the contract, and this article is noise.

Skip it if users never see long options. Internal argv glue does not belong in a changelog.

Skip it if you already snapshot prog --help and you freeze that snapshot in review. Help snapshots catch ghosts too, with a different failure shape. Use one method. Two methods that disagree will train people to ignore both.

Teams chasing writing quality scores should skip it. This protocol does not grade tone. It grades invented switches.

The merge rule I want

I merge a changelog section only when the newest heading exists, the ghost set is empty, and a human still rewrites the impact sentence.
The model may list flags. The parser must already own them.

Would you delete the bullet, or add the argument?
I delete the bullet unless a test also lands in the same PR. Flags without tests are how --json-pretty gets invented twice.

That is the whole review habit. Keep the extractor next to the CLI. Keep the failing fixture until the notes tell the truth.