DEV Community

Morgan Sun
Morgan Sun

Posted on

The Diff Is Draftable. The Support Window Is Not.

A services team cut a Friday release with a generated changelog that read clean. The bullets matched the merged commits. The tone was calm. The version bump in the same file said minor.

Monday, two mobile clients 404'd on a query parameter the server had stopped accepting. The model had summarized the diff. Nobody had classified the change. The support window was a sentence the generator invented because empty cells look unfinished.

That failure is not a writing problem. It is an ownership problem. Release docs mix two kinds of content that look similar in Markdown and are not similar in risk.

Two facts a model can use, and two it cannot

Git, OpenAPI diffs, and package manifests are evidence. A model can read them and draft user-facing summaries. SemVer, sunset dates, and “who still has to migrate” are contracts. Contracts need a named owner, not a fluent paragraph.

A model can usually see:

  • commit subjects and file paths in a tag range
  • added, removed, or renamed HTTP paths when you pass a schema diff
  • dependency name/version pairs from a lockfile
  • test names that started failing in CI

A model cannot honestly own:

  • whether the change is a major, minor, or patch for your compatibility promise
  • the last date an old field remains supported
  • who pages if a partner has not migrated
  • the rollback command that is safe in production, not the one that looks right in a README

Treat those as frozen fields. If the generator fills them, CI should fail closed.

Diff Lane vs Contract Lane

Split the release note into two lanes. Keep both in the same file so reviewers see them together. Do not let the draft lane write keys in the contract lane.

Diff Lane (model may draft)

  • changes[]: short, cited bullets tied to commit SHAs or schema paths
  • touched_paths[]: files or OpenAPI operations from the extractor, not from memory
  • dependency_deltas[]: old → new versions copied from the lockfile diff

Contract Lane (human must own)

  • semver_bump: major | minor | patch
  • breaking: boolean that must match the bump
  • support_until: an ISO date, or null only when breaking is false
  • migration_owner: a real handle in your org
  • rollback: a command or git tag that exists in the repo
  • signed_by: the reviewer who accepts the contract

The rule is simple. Summaries regenerate. Contracts persist across regenerations.

Worked example: a parameter removal

Label: proposed example, not a production incident dump.

Extractor output (facts only):

openapi_diff:
  removed: GET /v1/orders?include_archived
  added:   GET /v1/orders?archive_state=
commits:
  - 9f2c1aa  drop include_archived query flag
  - 41ab003  add archive_state enum filter
lockfile: no runtime dependency change
Enter fullscreen mode Exit fullscreen mode

A model may draft this Diff Lane:

## Changes (draft)
- Order listing no longer reads `include_archived`.
  Source: commit 9f2c1aa, OpenAPI removed query `include_archived`.
- Order listing accepts `archive_state`.
  Source: commit 41ab003, OpenAPI added query `archive_state`.
Enter fullscreen mode Exit fullscreen mode

A human must still fill this Contract Lane. Empty is better than guessed.

contract:
  semver_bump: major
  breaking: true
  support_until: "2026-12-31"
  migration_owner: "@orders-api"
  rollback: "git checkout 2026.09.12-orders && helm rollback orders 14"
  signed_by: "alex"
Enter fullscreen mode Exit fullscreen mode

If semver_bump is minor while a query parameter disappeared, the file is false even if every bullet is cited. Fluency does not make a compatibility promise.

Artifact: a freeze checker you can run in CI

Keep contract fields in YAML so a script can reject invented dates and unsigned bumps. The checker below is runnable as-is against a fixture. It does not call a model.

# freeze_changelog.py
# Proposed CI gate: validate Contract Lane. Does not generate prose.
from __future__ import annotations

import re
import sys
from datetime import date
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("pip install pyyaml\n")
    raise

BUMPS = {"major", "minor", "patch"}
HANDLE = re.compile(r"^@[A-Za-z0-9][A-Za-z0-9\-_/]*$")
ISO = re.compile(r"^\d{4}-\d{2}-\d{2}$")

FORBIDDEN_IN_CONTRACT = {
    "tbd",
    "todo",
    "soon",
    "n/a",
    "unknown",
    "the model",
    "as needed",
}


def load(path: Path) -> dict:
    data = yaml.safe_load(path.read_text(encoding="utf-8"))
    if not isinstance(data, dict) or "contract" not in data:
        raise SystemExit(f"{path}: missing top-level 'contract'")
    return data


def fail(errors: list[str]) -> None:
    if errors:
        sys.stderr.write("Contract Lane failed:\n")
        for item in errors:
            sys.stderr.write(f"  - {item}\n")
        raise SystemExit(1)


def check_contract(doc: dict) -> list[str]:
    c = doc["contract"] or {}
    errors: list[str] = []
    bump = c.get("semver_bump")
    breaking = c.get("breaking")
    until = c.get("support_until")
    owner = c.get("migration_owner")
    rollback = c.get("rollback")
    signed = c.get("signed_by")

    if bump not in BUMPS:
        errors.append("semver_bump must be major|minor|patch")
    if not isinstance(breaking, bool):
        errors.append("breaking must be a boolean")
    if bump == "major" and breaking is False:
        errors.append("major bump requires breaking: true")
    if bump in {"minor", "patch"} and breaking is True:
        errors.append("breaking: true requires major bump")

    if breaking:
        if not isinstance(until, str) or not ISO.match(until):
            errors.append("breaking changes need support_until as YYYY-MM-DD")
        else:
            try:
                parsed = date.fromisoformat(until)
            except ValueError:
                errors.append("support_until is not a real calendar date")
            else:
                if parsed <= date.today():
                    errors.append("support_until must be in the future")
    elif until not in (None, ""):
        errors.append("support_until must be null when breaking is false")

    for key in ("migration_owner", "rollback", "signed_by"):
        val = c.get(key)
        if not isinstance(val, str) or not val.strip():
            errors.append(f"{key} is empty")
        else:
            low = val.strip().lower()
            if any(tok in low for tok in FORBIDDEN_IN_CONTRACT):
                errors.append(f"{key} looks like a placeholder, not an owner")

    if isinstance(owner, str) and owner.strip() and not HANDLE.match(owner.strip()):
        errors.append("migration_owner must be an @handle")
    if not isinstance(signed, str) or signed.strip().startswith("@ai"):
        errors.append("signed_by must be a human reviewer")
    if isinstance(rollback, str) and "helm rollback" not in rollback and "git " not in rollback:
        errors.append("rollback must be an executable git or helm command")

    changes = doc.get("changes") or []
    if not changes:
        errors.append("Diff Lane is empty; extract commits before drafting")
    for item in changes:
        if not item.get("source"):
            errors.append("every drafted bullet needs a source SHA or schema path")
    return errors


def main(argv: list[str]) -> None:
    if len(argv) != 2:
        raise SystemExit("usage: python freeze_changelog.py path/to/release.yaml")
    path = Path(argv[1])
    fail(check_contract(load(path)))
    print(f"ok: {path}")


if __name__ == "__main__":
    main(sys.argv)
Enter fullscreen mode Exit fullscreen mode

Fixture that should pass:

# releases/2026-09-17.yaml
changes:
  - text: Order listing no longer reads include_archived.
    source: "commit:9f2c1aa"
  - text: Order listing accepts archive_state.
    source: "openapi:GET /v1/orders?archive_state"
contract:
  semver_bump: major
  breaking: true
  support_until: "2026-12-31"
  migration_owner: "@orders-api"
  rollback: "git checkout 2026.09.12-orders && helm rollback orders 14"
  signed_by: "alex"
Enter fullscreen mode Exit fullscreen mode

Run it:

pip install pyyaml
python freeze_changelog.py releases/2026-09-17.yaml
Enter fullscreen mode Exit fullscreen mode

Flip semver_bump to minor and run it again. The gate should refuse the file. That refusal is the documentation quality bar. A green bullet list is not.

Prompt boundary: fill Diff Lane only

Label: unexecuted prompt template. Pass extractor JSON in. Ask for YAML out. Name the keys the model must not emit.

You draft Diff Lane bullets only.
Input is JSON with commits[], openapi_diff, and lockfile_diff.
Write YAML with a top-level `changes` list.
Each item needs `text` and `source` copied from the input.
Do not emit `contract`, versions, dates, owners, or rollback commands.
If a fact is missing from input, omit the bullet. Do not guess.
Enter fullscreen mode Exit fullscreen mode

Pipe the result into the human-edited file with a merge that preserves contract: untouched. A three-line shell sketch:

# Proposed local workflow. Review before wiring to CI secrets.
python extract_release_facts.py v2026.09.12 HEAD > /tmp/facts.json
# model_draft.sh is your wrapper; it must write only `changes:`
model_draft.sh /tmp/facts.json > /tmp/diff_lane.yaml
python merge_lanes.py releases/2026-09-17.yaml /tmp/diff_lane.yaml
python freeze_changelog.py releases/2026-09-17.yaml
Enter fullscreen mode Exit fullscreen mode

merge_lanes.py should replace changes and leave contract byte-stable. If the model prints a support_until, drop the file. Do not parse it into the frozen map.

Decision table

Signal in the extractor Diff Lane may say Contract Lane must set
Query/path/field removed “X is removed”, with source breaking: true, major, future support_until
Field added, old field still served “Y is added” usually minor, breaking: false, support_until: null
Docs or comments only “docs updated”, if you even ship it patch
Lockfile patch of a transitive dep name and versions from the diff bump only if you promise that tree
CI test renamed nothing unless behavior changed do not bump from a rename
Empty extractor no bullets do not ship a changelog

The table is the review checklist. If a reviewer argues with the table, change the table in a docs PR. Do not special-case a single release in prose.

Where a hosted draft step fits

The freeze checker belongs in your repository. The draft step is the disposable part: it needs a model that can turn extractor JSON into changes[] and nothing else.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you do not want to stand up a private inference box for that draft job, MonkeyCode currently offers free model access and a free server option you can point the Diff Lane wrapper at. Keep Contract Lane signing on your CI runners either way. A hosted draft does not make a sunset date true.

Limitations

This workflow does not classify breaking changes for you. It only refuses unsigned or contradictory contracts. OpenAPI diffs miss behavior changes that never touch the schema: timeout shifts, pagination defaults, error payload shapes. Git subjects lie when people write “refactor” on a removal. The forbidden-token list is brittle; a model can emit a plausible date that is still wrong. The checker cannot see whether @orders-api is still the on-call rotation. rollback is a string match, not a dry-run against the cluster.

Do not treat a green freeze_changelog.py as a substitute for a staging consumer test. The script answers “did a human fill the contract keys.” It does not answer “will last quarter’s client still parse the response.”

Who should not use this

Skip the split if you ship a single private binary with no external clients and no compatibility promise. A personal changelog can stay a commit list. Skip it if legal or customer comms already owns a separate advisory and your repo notes are internal-only — duplicate sunset dates will drift. Skip it if your version numbers are calendar stamps with no SemVer meaning; then rename the bump field instead of pretending major applies. Teams that cannot name a migration_owner should stop generating release prose until ownership exists. A fluent draft without an owner is how Friday’s “minor” becomes Monday’s 404.

What to freeze next

After changelogs, the same split applies to deprecation banners in API reference pages and to “supported runtimes” tables in READMEs. The model may refresh the list of tested versions from CI badges. A human still owns the last date an old runtime is supported.

Start with one release YAML, one extractor, and the checker in CI. Leave the draft model interchangeable. The contract keys are the product.

Top comments (0)