Migration Diary: Fail Closed on Agent JSON Before You Leave the Paid Model

The silent failure in a model cutover is not a timeout, a 429, or a missing API key in staging. It is a JSON body that still validates, still logs cleanly, and still changes the meaning of a tool call. You can swap a paid runtime for a cheaper lane and keep HTTP 200 on every agent turn. Freeze the response contract first, then move traffic, or you will debug behavior that looks like application drift.

Why a green health check still ships a broken agent

Dashboards love latency, status codes, and token counts because those signals are cheap to scrape after every call. They will not tell you that priority became a string, or that steps lost the third item the UI still renders. Paid runtimes often wrap extras you never modeled, including citation blocks, safety tags, and vendor request identifiers. When you leave that runtime, those extras vanish, and your parser treats absence as a default the product never agreed to.

You should treat the JSON contract as a leased interface, the same way you treat a database schema during a blue-green cutover. The model remains an implementation detail, while the fields your tools and UI consume stay the public API. If you skip that distinction, the cutover will look finished while the leftovers keep executing in production paths.

1. Export the paid contract as fixtures, not as folklore

Do not start the migration by pointing a staging key at the new lane and clicking through the happy path. You need a frozen sample of real production shapes, including ugly ones, captured while the paid runtime is still authoritative. Those fixtures become the artifact you compare against, not a slide of typical output somebody remembered from last week.

Run the following proposed commands after you redact production logs and choose a fixture window you can defend.

mkdir -p fixtures/agent-output/{ok,partial,tool_error}
# Proposed extractor: replace with the store that already keeps raw model bodies.
python scripts/export_raw_bodies.py \
  --since 2026-08-01 \
  --out fixtures/agent-output \
  --strip-secrets

Keep at least three buckets because those shapes diverge first after a model swap: strict success, partial tool loops, and explicit tool errors. Redact API keys, user content, and account identifiers before any fixture leaves the production network. Then write a small inventory that names every field the application actually reads, not every field the model happened to emit.

# fixtures/contract.yml  (proposal)
version: 1
required_object: agent_turn
fields:
  - name: task_id
    type: string
    consumed_by: orchestrator
  - name: status
    type: enum
    values: [ok, needs_input, failed]
    consumed_by: ui
  - name: tool_calls
    type: array
    consumed_by: dispatcher
  - name: citations
    type: array
    optional: true
    consumed_by: none  # leftover from the paid wrapper

Mark consumed_by: none leftovers in writing before anyone deletes the paid account. Those fields are the ones you must stop branching on, because they will not travel with you. A field nobody owns is not optional flavor; it is an untracked dependency sitting in a parser.

2. Freeze a validator that fails closed

A schema that only warns in logs will not protect a dispatcher that is already holding a tool name. The comparator has to reject the turn before any side effect runs, including retries that look harmless in staging. Fail closed on unknown keys, missing idempotency keys, and status values the UI cannot render.

The following Python is an unexecuted example you can adapt; it is a proposal, not a benchmarked library.

# scripts/validate_turn.py  (proposal)
from jsonschema import Draft202012Validator
import json, sys, pathlib

SCHEMA = {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "additionalProperties": False,
    "required": ["task_id", "status", "tool_calls"],
    "properties": {
        "task_id": {"type": "string", "minLength": 1},
        "status": {"enum": ["ok", "needs_input", "failed"]},
        "tool_calls": {
            "type": "array",
            "items": {
                "type": "object",
                "additionalProperties": False,
                "required": ["name", "arguments", "idempotency_key"],
                "properties": {
                    "name": {"type": "string", "pattern": "^[a-z][a-z0-9_]*$"},
                    "arguments": {"type": "object"},
                    "idempotency_key": {"type": "string", "minLength": 8},
                },
            },
        },
        "citations": {"type": "array"},
    },
}

def validate(path: str) -> None:
    payload = json.loads(pathlib.Path(path).read_text())
    Draft202012Validator(SCHEMA).validate(payload)

if __name__ == "__main__":
    validate(sys.argv[1])
    print("ok", sys.argv[1])

Notice additionalProperties: False and the required idempotency_key on every tool call. Extra vendor wrappers fail loudly instead of sliding into a loosely typed dictionary that your dispatcher later trusts. Missing keys fail before a tool runs twice, which is the actual incident you are trying not to replay.

Run the validator against the fixtures you exported, including the ugly ones that never appear in demo scripts.

find fixtures/agent-output -name '*.json' -print0 \
  | xargs -0 -n1 python scripts/validate_turn.py

If paid-runtime fixtures fail this schema, you do not have a migration problem yet. You have an undocumented parser, and you should fix that parser before any lane swap. A cutover that starts from folklore will only freeze the folklore.

3. Dual-run the candidate lane against the same prompts

Only after fixtures pass should you introduce a second implementation behind the same schema. Keep the paid runtime as the primary writer so production meaning does not move yet. Send a shadow copy of the prompt to the candidate lane, then compare both bodies with the same validator and a field-level diff.

# scripts/shadow_compare.py  (proposal)
import json, hashlib
from pathlib import Path

IGNORE = {"citations"}  # leftovers you already decided not to consume

def load(path: str) -> dict:
    return json.loads(Path(path).read_text())

def fingerprint(obj: dict) -> str:
    cleaned = {k: v for k, v in obj.items() if k not in IGNORE}
    blob = json.dumps(cleaned, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()[:12]

def compare(paid_path: str, candidate_path: str) -> dict:
    paid, cand = load(paid_path), load(candidate_path)
    return {
        "paid_fp": fingerprint(paid),
        "cand_fp": fingerprint(cand),
        "status_match": paid.get("status") == cand.get("status"),
        "tool_name_match": [t["name"] for t in paid.get("tool_calls", [])]
            == [t["name"] for t in cand.get("tool_calls", [])],
        "candidate_only_keys": sorted(set(cand) - set(paid)),
        "paid_only_keys": sorted(set(paid) - set(cand)),
    }

Store every mismatch as a leftover, not as a vague quality complaint in chat. Candidate-only keys are usually new wrappers the new lane added without asking. Paid-only keys are usually vendor extras your UI still reads by accident, which is how silent defaults get invented.

Use the table below when the comparator is noisy and you need a consistent rule instead of a meeting.

Comparator signal	Treat it as	Cutover action
Schema failure on candidate	Contract break	Keep paid primary; do not flip
paid_only_keys that consumed_by is none	Wrapper leftover	Strip in adapter; do not re-encode
paid_only_keys that UI still reads	Hidden dependency	Add to schema or remove the branch
candidate_only_keys	New wrapper	Reject until schema is revised
status_match true, tool_name_match false	Side-effect risk	Human review before any drain
Fingerprints match after IGNORE	Shape-compatible	Eligible for a limited shadow sample

This is the moment a free shadow lane is useful, because dual-running should not inflate the paid bill while you collect diffs. MonkeyCode's free model access and free server option can host that shadow lane while the validator and the fixture set stay in your repository.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Do not treat that lane as production yet, and do not paste live customer prompts into it without the same redaction rules you used for fixtures. Point only the comparator at it, drop any response that fails the frozen schema, and keep the paid writer in front of every tool dispatcher.

4. Drain in-flight turns before you flip the default route

A schema freeze is not a cutover, even when the comparator looks boring. You still have turns that started on the paid runtime and will finish after you change the default. Mixed-schema resumes are the leftover that creates duplicate side effects, because each lane mints a different idempotency_key for the same user intent.

Follow this drain plan in order, and do not skip a step because staging looked quiet.

1. Stop admitting new long-running agent jobs to the paid model once the comparator is clean for a full fixture pass.
2. Let in-flight turns finish on the original runtime so partial tool_calls arrays are never mixed across vendors.
3. Reject any resume payload whose task_id was minted before the freeze if the schema version does not match.
4. Flip the default route only for new task_id values, and keep the paid lane as a manual rollback for one drain window.
5. Re-run the fixture suite against production logs the next morning, not against the lab samples you already memorized.

Gate resumes in the orchestrator so a leftover task cannot quietly continue on the wrong lane.

# proposal: refuse mixed-schema resumes
CURRENT_SCHEMA = 1
CURRENT_LANE = "candidate"

def can_resume(task) -> bool:
    if task.schema_version != CURRENT_SCHEMA:
        return False
    if task.model_lane != CURRENT_LANE and task.state != "completed":
        return False
    return True

If you skip the drain, you will see the worst leftover on the first busy morning. A tool will run twice, once per lane, and your logs will show two valid JSON bodies for one user action. Validity is not safety when both bodies satisfy the schema and neither body knows about the other.

Leftovers that survive a green validator

A frozen schema catches shape. It does not catch tone, citation honesty, or a tool name that is legal but unwise. You will still find truncated lists that satisfy type: array, enums that cluster on the first value, and arguments that are empty objects. Keep a human review queue for mismatches where status_match is true and tool_name_match is false, because that is how silent side effects start.

Write the leftovers down before you delete the paid account, while the vendor console is still there to copy from. Streaming trailers may still be concatenated into a JSON body by an old client. Vendor request identifiers may still be baked into log dashboards and support runbooks. Default temperatures and stop sequences may still live in a console, not in git. Retry maps may still assume a paid rate-limit header you will not see on the new lane. Prompt snippets may still tell the old model to emit a wrapper key you have now forbidden.

Put those notes next to contract.yml so the next person does not reintroduce them as helpful extras. A leftover that is documented is a migration task. A leftover that is only remembered is the next incident report.

Limitations, and who should not use this

This workflow assumes you can store raw model bodies and that your product already consumes structured output. If your interface is free-form chat with no tool dispatcher, a JSON freeze will add ceremony without reducing risk. If you cannot redact fixtures, you should not dual-run anything that contains user text, because the shadow lane then becomes a second copy of private prompts.

The approach also does not prove quality, and a candidate can pass the schema while still being a worse assistant. Free model access and a free server do not make the schema optional, and they do not guarantee identical tool routing. Skip this cutover if you cannot fail closed, cannot keep the paid lane up for a drain window, or cannot review tool-name mismatches by hand.

Regulated workflows that forbid shadow copies of prompts should stop at fixture export and legal review. Do not invent a dual-run just because the validator is ready. The contract freeze still helps those teams, but the shadow lane does not.

The morning after the flip

Read the comparator report before you read the cost report, because cheap wrong JSON is still wrong JSON. If candidate-only keys appeared overnight, extend the schema or strip the wrapper; do not let additionalProperties drift back to true. If paid-only keys disappeared and a UI branch still mentioned them, that branch is now a leftover default and belongs on the same list as the vendor wrappers.

If the fixture suite already sits in CI, the useful next step is a non-production shadow lane on free model access, confirming the validator still fails closed before any second drain.