If the Shape Changed, the PR Failed: A Compatibility Checklist

AI-assisted PRs usually die in production on a shape change, not on a red unit test. If a field vanished, a type narrowed, or a required key appeared, treat the pull request as failed even when coverage is green. Compatibility is a merge gate. It is not a code-review comment.

You already know the cheap-code pattern. A model rewrites a handler in minutes. Tests assert status 200. Staging looks calm. Then an old mobile client, a batch consumer, or a warehouse job reads yesterday's payload and throws. The bug is not logic. The bug is the contract.

This article is a copyable fail-closed checklist for JSON APIs and event payloads. Use it when a model touches serializers, OpenAPI files, protobuf JSON mappings, or “small cleanup” refactors. The artifact is a schema-diff script you can run locally or in CI. No personal production war stories here. The workflow is proposed and labeled as such.

Why green tests still ship breaking clients

Unit tests are written against the new fixture. That is the trap. The model updates the sample JSON, the assertion, and the handler in the same diff. The suite agrees with itself. Downstream systems never vote.

Cheap generation makes this worse. When code is inexpensive, teams merge more surface area per day. Unowned modules accumulate. Payload keys drift. Nobody owns the old shape, so nobody notices it left.

You do not need a platform team to stop the cheap ones. You need a gate that fails closed on incompatible shapes and asks for evidence when a change is allowed.

Change taxonomy: allow, wrap, or reject

Copy this table into the PR template. If the change is not in the allow column, the default is reject.

Change	Typical AI rationale	Merge default	Evidence required
Add optional field	“clients might want this”	Allow	Schema diff + sample with field absent
Add new endpoint / event type	“cleaner split”	Allow	Old path still served; dual-route test
Widen a type (int → number, extra enum)	“more flexible”	Allow	Parser test on old and new values
Remove field	“unused”	Reject	None. Keep field or version the API
Rename field	“clearer name”	Reject	Alias both names for one release
Make field required	“validation”	Reject	Default or backfill, then optional
Narrow type / drop enum	“stricter”	Reject	New type on a new field
Tighten additionalProperties	“lock the object”	Reject	Unknown keys must still round-trip
Change nullability	“the value is always set”	Reject	Dual-read both null and value
Reorder-only JSON object	“style”	Allow	Prove consumers are key-based

Fail closed means: missing evidence is a no, not a maybe. “We think no client uses it” is not evidence.

Fail-closed criteria you can paste

Use these as CI exits, not as review nits.

1. Baseline exists. schema/old.json is the last released schema, not the schema the model just invented.
2. Diff is classified. Every changed pointer is tagged additive, breaking, or unknown. unknown fails the build.
3. Removed pointers fail. Deleted properties, dropped enum values, and vanished union members are breaking.
4. New required keys fail. A required field that old producers omit is breaking.
5. Type narrowing fails. string to uuid, number to integer, or a oneOf member disappearing is breaking.
6. Consumers still parse the old body. Replay at least one captured production payload (redact secrets first) against the new parser.
7. Producers still emit the old body if you have mixed-version clients. New fields must be optional for one release.
8. Version bump is explicit. Breaking changes require a new path, event name, or schema version. Silent v1 mutation fails.

If any item cannot run, the gate stays closed. Skipping the gate because “this is just a refactor” is how refactors ship outages.

Artifact: compat_gate.py

Proposed workflow, unexecuted against your repo until you point it at real files. Save this next to two snapshots: schema/released.json and schema/pr.json.

#!/usr/bin/env python3
"""Fail closed on JSON Schema breaking changes. Python 3.10+."""
from __future__ import annotations

import json
import sys
from pathlib import Path
from typing import Any

BREAKING = 0

def load(path: str) -> dict[str, Any]:
    data = json.loads(Path(path).read_text())
    if not isinstance(data, dict):
        raise SystemExit(f"schema must be an object: {path}")
    return data

def props(schema: dict[str, Any]) -> dict[str, Any]:
    return dict(schema.get("properties") or {})

def required(schema: dict[str, Any]) -> set[str]:
    return set(schema.get("required") or [])

def enum_vals(node: dict[str, Any]) -> set[Any] | None:
    if "enum" not in node:
        return None
    return set(node["enum"])

def fail(msg: str) -> None:
    global BREAKING
    BREAKING += 1
    print(f"BREAKING: {msg}")

def warn(msg: str) -> None:
    print(f"ADDITIVE: {msg}")

def check_node(ptr: str, old: dict[str, Any], new: dict[str, Any]) -> None:
    old_type, new_type = old.get("type"), new.get("type")
    if old_type and new_type and old_type != new_type:
        fail(f"{ptr} type {old_type!r} -> {new_type!r}")
    old_enum, new_enum = enum_vals(old), enum_vals(new)
    if old_enum is not None and new_enum is not None:
        dropped = old_enum - new_enum
        if dropped:
            fail(f"{ptr} dropped enum values {sorted(dropped, key=str)}")
        added = new_enum - old_enum
        if added:
            warn(f"{ptr} added enum values {sorted(added, key=str)}")
    if old.get("additionalProperties") is True and new.get("additionalProperties") is False:
        fail(f"{ptr} additionalProperties true -> false")
    old_props, new_props = props(old), props(new)
    for name in sorted(set(old_props) - set(new_props)):
        fail(f"{ptr}.{name} removed")
    for name in sorted(set(new_props) - set(old_props)):
        warn(f"{ptr}.{name} added")
    for name in sorted(set(old_props) & set(new_props)):
        o, n = old_props[name], new_props[name]
        if isinstance(o, dict) and isinstance(n, dict):
            check_node(f"{ptr}.{name}", o, n)
    extra_req = required(new) - required(old)
    if extra_req:
        fail(f"{ptr} new required fields {sorted(extra_req)}")

def main() -> None:
    if len(sys.argv) != 3:
        raise SystemExit("usage: compat_gate.py released.json pr.json")
    old, new = load(sys.argv[1]), load(sys.argv[2])
    check_node("$", old, new)
    print(f"breaking_count={BREAKING}")
    raise SystemExit(1 if BREAKING else 0)

if __name__ == "__main__":
    main()

Run it like this:

python3 compat_gate.py schema/released.json schema/pr.json
echo $?

A zero exit is the only merge signal. Non-zero means the shape changed in a way old clients cannot ignore. Do not “fix” a red gate by editing released.json in the same PR. That deletes the baseline.

Minimal fixtures so you can see both exits

schema/released.json:

{
  "type": "object",
  "required": ["id", "status"],
  "additionalProperties": true,
  "properties": {
    "id": {"type": "string"},
    "status": {"type": "string", "enum": ["open", "closed", "held"]},
    "note": {"type": "string"}
  }
}

schema/pr.json (this should fail):

{
  "type": "object",
  "required": ["id", "status", "assignee"],
  "additionalProperties": false,
  "properties": {
    "id": {"type": "string"},
    "status": {"type": "string", "enum": ["open", "closed"]},
    "assignee": {"type": "string"}
  }
}

Expected report:

BREAKING: $.note removed
BREAKING: $.status dropped enum values ['held']
BREAKING: $ additionalProperties true -> false
BREAKING: $ new required fields ['assignee']
ADDITIVE: $.assignee added
breaking_count=4

Wire it in CI so a model cannot skip it:

set -euo pipefail
test -f schema/released.json
python3 compat_gate.py schema/released.json schema/pr.json

If schema/pr.json is generated from code, generate it in CI from the PR branch, never from a prompt transcript. The file in git is the contract. Chat output is not.

Evidence block for the PR

Paste this under the diff. Empty fields fail the review, same as a red gate.

## Compatibility evidence
- Released schema commit: 
- `compat_gate.py` exit: 0 / 1 (attach log)
- Breaking pointers: none | list
- Old production payload replayed: yes / no (redacted fixture path)
- Mixed-version clients in the wild: none | iOS n-1 | worker X
- If breaking: new path or schema version:
- Rollback: old binary still serves the old shape? yes / no

Rollback is part of compatibility. If you cannot run the previous artifact and still emit the released shape, you do not have a rollback. You have a hope.

Where a free model and a free server actually help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The model is a draft engine. It is not the compatibility authority. Ask it to emit schema/pr.json from the handler diff, then throw that file at compat_gate.py. If the gate is red, the PR is red. Do not ask the same model to “confirm the change is backward compatible.” That is grading its own exam.

MonkeyCode's free model access and free server option are enough to host this loop: generate the candidate schema, run the gate, replay one redacted fixture. Keep secrets off that box. Keep released.json copied from the last tagged release, not from the model's memory.

Use the model for additive work only after the gate is green: optional fields, new event types, documentation of the old shape. If you want to try that split on a throwaway service, MonkeyCode is one place you can run the script without standing up a private runner first.

Limitations

This checker is shallow on purpose. It understands object properties, required, enum, type, and additionalProperties. It does not understand $ref graphs, allOf / oneOf normalization, defaulting, HTTP header contracts, or SQL migrations. Nested arrays of unions will need a real JSON Schema library.

It also cannot see semantic breaks. Same types, different meaning (status: "open" now means “draft”) will pass. Pair this gate with one captured payload replay, not with vibes.

JSON key order is ignored. If you have a consumer that parses payloads as ordered tuples, this workflow will not save you. Fix that consumer.

Who should not use this

Skip the gate if you ship a single binary to a single client you control and you deploy them together. A native desktop app with a matching server cut can change shapes in lockstep. The checklist is for mixed-version clients, public APIs, event buses, and anything a warehouse still reads.

Do not use it as a substitute for auth, threat modeling, or load tests. A compatible payload can still leak PII or melt a connection pool.

Do not point the gate at a schema the model wrote for both sides of the diff. Without a released baseline, fail closed by doing nothing else: refuse the merge.

Close the loop

Start with one endpoint. Freeze schema/released.json from production, not from main if main has drifted. Run the script on the next AI PR that “just cleans up naming.” If it fails, that is the process working. Cheap code is fine. Cheap contracts are not.