Pin CLI JSON Keys Before an Agent Renames Them

Tuesday's digest folder filled with empty files. The summarizer CLI had exited 0 every night. Cron still ran jq -r .excerpt. The field no longer existed.

A Sunday agent pass had renamed excerpt to blurb for readability. Unit tests still passed. They checked that stdout parsed as a dict and that the process code was zero. Nobody had pinned the keys a wrapper already consumed.

This write-up is a lab fixture, not a production postmortem. The clocks below are time boxes, not fleet metrics. The work is one frozen JSON contract for a tiny --format json CLI, plus a local checker that exits 1 when an agent shuffles keys.

Cron remembers the old name

Wrappers are stubborn in a way dashboards are not. A crontab line written six months ago still asks for .excerpt. A GitHub Action still pipes the same jq filter. The agent never sees those callers. It sees a Python file and a README that can be rewritten in one diff.

Cheap generation makes that rewrite feel free. The debt shows up later, in a job that writes zero-byte artifacts and still reports success. Exit code 0 is not a document of shape.

Think of a labeled kitchen drawer. The CLI used to keep the scissors under excerpt. Someone moved the scissors and wrote blurb on the front. The night job still opens excerpt, then files the emptiness as a completed task.

Keys are the public surface

For a small JSON CLI, the public surface is the key set, the types, and the extra-key policy. Help text is commentary. Docstrings are commentary. A test that only calls json.loads is commentary with a green checkmark.

The contract here is a JSON file checked into the repo. It lists required keys, their types, and whether unknown keys are allowed. The checker loads one real CLI payload and compares. It does not call a network model. It does not crawl GitHub Actions. It only answers whether tonight's object still matches the keys a wrapper already parses.

Forty-five minutes and zero dollars is enough for this cut. If the script grows past one file plus one schema, stop. Rollback is deleting schemas/cli_json_v1.json and scripts/check_json_contract.py.

A CLI that can drift on purpose

Save this as app.py. The default path emits excerpt. A second branch, flipped by an environment variable, emits blurb so the break can be reproduced without waiting for a real agent patch.

#!/usr/bin/env python3
"""Minimal summarizer CLI with a JSON mode that can drift."""

from __future__ import annotations

import json
import os
import sys
from pathlib import Path


def summarize(source: Path) -> dict:
    text = source.read_text(encoding="utf-8")
    payload = {
        "source": str(source),
        "bytes": source.stat().st_size,
        "excerpt": text.strip()[:120],
    }
    if os.environ.get("CLI_JSON_DRIFT") == "1":
        payload["blurb"] = payload.pop("excerpt")
    return payload


def main() -> int:
    if "--format" not in sys.argv or "json" not in sys.argv:
        print("usage: app.py --format json --source <file>", file=sys.stderr)
        return 2
    try:
        src_flag = sys.argv.index("--source")
        source = Path(sys.argv[src_flag + 1])
    except (ValueError, IndexError):
        print("missing --source", file=sys.stderr)
        return 2
    if not source.is_file():
        print("source not found", file=sys.stderr)
        return 2
    json.dump(summarize(source), sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    sys.exit(main())

A fixture file keeps the demo honest. Save fixtures/app.log with two short log lines. The exact wording does not matter. The keys do.

One versioned contract file

Save this as schemas/cli_json_v1.json. Version the filename. Do not overwrite v1 in place when a key is retired. That is how wrappers get a grace period instead of a surprise.

{
  "name": "summarizer-json-v1",
  "additionalProperties": false,
  "required": ["source", "bytes", "excerpt"],
  "properties": {
    "source": {"type": "string", "minLength": 1},
    "bytes": {"type": "integer", "minimum": 0},
    "excerpt": {"type": "string"}
  }
}

additionalProperties set to false is the sharp edge. Agents love adding confidence, model, or notes. Those extra fields look harmless until another tool serializes the whole object into a store that rejects unknown columns. Surprise keys count as a break.

A checker with no extra packages

Save this as scripts/check_json_contract.py. Standard library only. No JSON Schema package, no network, no git inspection.

#!/usr/bin/env python3
"""Exit 1 when CLI JSON drifts from a frozen key contract."""

from __future__ import annotations

import json
import sys
from pathlib import Path
from typing import Any


def load_json(path: Path) -> Any:
    return json.loads(path.read_text(encoding="utf-8"))


def type_ok(value: Any, expected: str) -> bool:
    mapping = {
        "string": str,
        "integer": int,
        "number": (int, float),
        "boolean": bool,
        "object": dict,
        "array": list,
    }
    if expected == "integer" and isinstance(value, bool):
        return False
    wanted = mapping[expected]
    return isinstance(value, wanted)


def check(schema: dict, payload: Any) -> list[str]:
    errors: list[str] = []
    if not isinstance(payload, dict):
        return ["payload is not an object"]
    required = schema.get("required", [])
    properties = schema.get("properties", {})
    extras_ok = schema.get("additionalProperties", True)
    for key in required:
        if key not in payload:
            errors.append(f"missing required key: {key}")
    for key, value in payload.items():
        spec = properties.get(key)
        if spec is None:
            if not extras_ok:
                errors.append(f"undeclared key: {key}")
            continue
        expected = spec.get("type")
        if expected and not type_ok(value, expected):
            errors.append(
                f"{key}: expected {expected}, got {type(value).__name__}"
            )
        if expected == "string" and spec.get("minLength") and len(value) < spec["minLength"]:
            errors.append(f"{key}: empty string")
        if expected == "integer" and "minimum" in spec and value < spec["minimum"]:
            errors.append(f"{key}: {value} below minimum {spec['minimum']}")
    return errors


def main() -> int:
    if len(sys.argv) != 3:
        print(
            "usage: check_json_contract.py <schema.json> <payload.json>",
            file=sys.stderr,
        )
        return 2
    schema = load_json(Path(sys.argv[1]))
    payload = load_json(Path(sys.argv[2]))
    errors = check(schema, payload)
    if errors:
        print("JSON contract failed:")
        for item in errors:
            print(f"  - {item}")
        return 1
    print(f"JSON contract ok: {schema.get('name', 'unnamed')}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The checker never executes the agent. It never grades the quality of the summary. It only compares shapes. Shape is what cron actually consumes.

Reproduce the rename

Create a short log, emit JSON, and validate the happy object.

mkdir -p fixtures schemas scripts
printf '%s\n' 'ERROR disk 91%' 'WARN retry 3' > fixtures/app.log
python3 app.py --format json --source fixtures/app.log > /tmp/cli.json
python3 scripts/check_json_contract.py schemas/cli_json_v1.json /tmp/cli.json
echo $?

A healthy run prints JSON contract ok: summarizer-json-v1 and exits 0. Then flip the drift switch the way an agent rename would.

CLI_JSON_DRIFT=1 python3 app.py --format json --source fixtures/app.log > /tmp/cli-drift.json
python3 scripts/check_json_contract.py schemas/cli_json_v1.json /tmp/cli-drift.json
echo $?
cat /tmp/cli-drift.json

Expected lines from that second run:

JSON contract failed:
  - missing required key: excerpt
  - undeclared key: blurb

Exit code 1. That is the whole alarm. The CLI still exited 0. The contract did not.

A type break is worth a third shot. Hand-edit /tmp/cli.json so bytes becomes "412" with quotes, then run the checker again. The report should mention bytes: expected integer, got str. Agents stringify fields for safety. Wrappers that later add the field then concatenate instead of counting.

Draft the contract off the laptop

A sample payload is enough to start the file. Some solo builders paste that payload into a remote draft box so the laptop never holds a model runtime.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project with free model access and a free server option. Operator notes for this account also include a 10-million-token free allowance on that drafting path. Those are availability notes, not hardware specs, not latency numbers, not a promise that any given draft will be correct. The contract still has to land in git and run locally against a payload the model never saw.

The safe loop is narrow. Capture one real --format json object on the machine that already runs the CLI. Sanitize paths and log text. Draft cli_json_v1.json on the free server from that object. Copy the file back. Run check_json_contract.py on a second payload. If the draft invents optional keys the CLI does not emit, delete those keys before committing. Remote drafts are generous with extras. Wrappers are not.

Keep private logs off the remote side. Fake the source string. Fake the excerpt. Key names are the only thing the contract needs.

Pass and fail at a glance

Payload vs contract	Exit	Why it matters
Exact required keys, right types	0	Wrapper still parses
Required key renamed	1	Cron still queries the old name
Extra key, additionalProperties false	1	Surprise field leaks downstream
Extra key, additionalProperties true	0	Only during a staged alias window
bytes sent as a string	1	jq math and Python addition both lie
Empty JSON array	1	The contract is one object
CLI process 0, contract 1	fail the change	Process success is not shape success

Paste that table into the pull request when the CLI is being improved. Reviewers can argue about aliases. They should not argue about whether .excerpt still exists.

Lies the checker will not catch

The script does not read English. An excerpt that is always the string TODO will pass. A source path that points at a file the wrapper should never see will pass if the type is string. Semantic lies need a different fixture.

It does not handle NDJSON streams, pretty-print drift inside string values, or locale-formatted numbers. It does not version HTTP APIs. It does not replace consumer-driven contracts for a fleet of services.

Unknown-key policy can be gamed. Setting additionalProperties to true for now is how the freeze dies. If a new key is intentional, cut cli_json_v2.json and move wrappers on a schedule.

Boolean-as-integer is handled. Union types are not. If a field is null on empty logs, say so in a later file. Do not hide nulls by deleting the key. Missing keys and nulls are different to jq.

Skip this freeze

Leave the files unwritten when the CLI has no JSON mode. Leave them unwritten when a team already publishes OpenAPI or JSON Schema from the same code that emits the object. Leave them unwritten for pretty TTY output meant for humans. A contract will only punish help text.

Also skip it for one-off notebooks and for binaries whose output is a file on disk, not stdout. This ritual assumes a wrapper that reads stdout and stays quiet when a key vanishes.

One-release aliases, then cut v2

If a wrapper still keys off excerpt while the CLI prefers blurb, emit both for one tagged release. Put both names in v1 with additionalProperties still false. Then ship v2 that drops excerpt, and move the crontab in the same change. Two releases. Not a forever alias bag.

Drop the ritual if schema edits take longer than the CLI change, if someone starts generating the contract from the same agent that edits app.py, or if the checker is taught to warn instead of exit 1. A warning on JSON drift is how empty digest folders return.

The schema file is the afternoon experiment. Point it at one --format json command. Keep the wrapper's key list even if the Python script gets deleted. The keys were always the product.

If a wrapper still reads a retired field, write down the old name, the new name, and the release that will drop the alias. That triple is enough to plan the crontab change. Stop there.