The Agent Invented a status_reason. The Spec Had Never Listed It.

Have you ever watched a generated HTTP client look finished, then fail the first real fixture you threw at it? I spent forty-eight hours in that loop last week, and the failure mode stayed almost polite. Every request returned 200, every dataclass carried a type hint, and every mock-based test stayed green. The only bug was a field the upstream API had never promised, and I kept blaming the transport.

This is a field notebook, not a product tour. I am writing down what I tried, what broke, and what I would actually repeat. The reduced example below is a local reproduction I kept, not a production incident report.

Hour 0–8: I asked for a client, not a contract

I dropped a short docstring into a coding session and asked for a Python adapter around a billing events endpoint. The prompt named the route, mentioned pagination, and said almost nothing about the JSON body. Why would I paste a schema when the agent could “just read the handlers,” right?

What I tried, in order:

1. I asked for a typed client with retries, timeouts, and a BillingEvent dataclass that “matches the API.”
2. I asked it to invent pytest mocks so I could keep iterating without hitting the sandbox.
3. I asked it to “fill any remaining fields from common billing platforms” when the first draft looked thin.
4. I grepped the generated module for status and assumed a 200 path meant the payload shape was honest.

The generated adapter looked like this. Notice the extra keys. I did not notice them for a long time:

# generated_client.py — reduced from the first draft I accepted
from dataclasses import dataclass
from typing import Any
import json, urllib.request

@dataclass
class BillingEvent:
    id: str
    amount_cents: int
    currency: str
    status: str
    status_reason: str          # invented
    processor_ref: str          # invented
    risk_score: float           # invented
    created_at: str

class BillingClient:
    def __init__(self, base: str, token: str) -> None:
        self.base = base.rstrip("/")
        self.token = token

    def get_event(self, event_id: str) -> BillingEvent:
        req = urllib.request.Request(
            f"{self.base}/v1/billing/events/{event_id}",
            headers={"Authorization": f"Bearer {self.token}"},
        )
        with urllib.request.urlopen(req, timeout=10) as resp:
            payload: dict[str, Any] = json.load(resp)
        return BillingEvent(**payload)  # this is the landmine

Does that look complete to you? It looked complete to me. The type hints were confident, and the happy-path tests were even more confident because they used the agent’s own mocks.

Hour 8–24: What actually broke

The first captured fixture I dropped in was tiny. Four keys. No status_reason. No processor_ref. No risk_score. The live call still returned HTTP 200, which is how I wasted a day on timeouts, User-Agent headers, and TLS.

{
  "id": "evt_9f3a",
  "amount_cents": 4200,
  "currency": "usd",
  "status": "posted",
  "created_at": "2026-09-05T18:11:04Z"
}

What broke, in the order I discovered it:

• BillingEvent(**payload) raised TypeError on missing invented fields, so I “fixed” it with defaults.
• Defaults made every test green again, including tests that never opened the fixture file.
• A later dump of asdict(event) wrote risk_score: 0.0 into an audit log that a reviewer treated as real.
• I spent hours comparing two JSON files that disagreed only because one of them was generated from the dataclass, not from the wire.

The agent had not crashed. The agent had been helpful in the most expensive way: it completed a shape I never specified. Have you noticed how often a 200 response makes us skip the only question that matters? What keys did the server actually send?

I finally stopped staring at urllib and wrote the smallest contract check I could keep in the repo.

The artifact: a four-file contract loop

This is the reproduction I would keep even if I threw the client away. Four files. No framework. The point is to fail before a dataclass can invent gravity.

1. contracts/billing_event.schema.json — the only source of truth I will paste into a prompt:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://example.local/contracts/billing_event.schema.json",
  "title": "BillingEvent",
  "type": "object",
  "additionalProperties": false,
  "required": ["id", "amount_cents", "currency", "status", "created_at"],
  "properties": {
    "id": {"type": "string", "pattern": "^evt_"},
    "amount_cents": {"type": "integer", "minimum": 0},
    "currency": {"type": "string", "enum": ["usd", "eur"]},
    "status": {"type": "string", "enum": ["posted", "void", "pending"]},
    "created_at": {"type": "string", "format": "date-time"}
  }
}

2. fixtures/evt_9f3a.json — a captured body, not a mock the agent wrote for itself.

3. tests/test_billing_contract.py — validate the fixture, then validate any client dump:

# tests/test_billing_contract.py
from pathlib import Path
import json
import pytest
from jsonschema import Draft202012Validator

ROOT = Path(__file__).resolve().parents[1]
SCHEMA = json.loads((ROOT / "contracts/billing_event.schema.json").read_text())
FIXTURE = json.loads((ROOT / "fixtures/evt_9f3a.json").read_text())
VALIDATOR = Draft202012Validator(SCHEMA)

def test_captured_fixture_matches_schema():
    errors = sorted(VALIDATOR.iter_errors(FIXTURE), key=lambda e: e.path)
    assert errors == [], [e.message for e in errors]

@pytest.mark.parametrize("banned", ["status_reason", "processor_ref", "risk_score"])
def test_schema_rejects_invented_keys(banned):
    bloated = dict(FIXTURE)
    bloated[banned] = "nope"
    errors = list(VALIDATOR.iter_errors(bloated))
    assert any("additional" in e.message.lower() or banned in str(e.path) for e in errors)

def test_client_dump_must_be_a_subset_of_the_spec(tmp_path):
    # Pretend the adapter wrote an audit blob. Feed it back through the schema.
    dump = tmp_path / "event.dump.json"
    dump.write_text(json.dumps(FIXTURE))
    body = json.loads(dump.read_text())
    Draft202012Validator(SCHEMA).validate(body)

4. Makefile — the commands I actually reran instead of rereading the dataclass:

.PHONY: contract
contract:
    python -m pytest tests/test_billing_contract.py -q
    python -c "from pathlib import Path; import json; from jsonschema import Draft202012Validator;\
 s=json.loads(Path('contracts/billing_event.schema.json').read_text());\
 Draft202012Validator.check_schema(s); print('schema ok')"

Run it once against the invented client dump and the failure is loud. Run it against the captured fixture and the suite is boring, which is the entire point. Would I rather debug TypeError in a dataclass constructor, or a schema error that names the extra key?

Hour 24–48: I regenerated against the spec, not the docstring

The second pass only worked after I changed the prompt ingredients. I did not ask for “a complete billing client.” I pasted billing_event.schema.json, forbade additionalProperties, and said the dataclass fields must equal required plus properties, nothing else.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode’s free model access and free server option for that regeneration pass so the new tree was not mixed with the first draft’s mocks. I am not claiming a model name, a quota, a hardware profile, or a lasting free tier. I am claiming a workflow: spec in, client out, fixture as the judge.

The second client was smaller and slightly embarrassing:

@dataclass
class BillingEvent:
    id: str
    amount_cents: int
    currency: str
    status: str
    created_at: str

ALLOWED = set(BillingEvent.__dataclass_fields__)

def from_payload(payload: dict) -> BillingEvent:
    extra = set(payload) - ALLOWED
    missing = ALLOWED - set(payload)
    if extra or missing:
        raise ValueError(f"contract drift extra={extra!r} missing={missing!r}")
    return BillingEvent(**payload)

That ValueError is the whole lesson. If the server grows a field, I want a failed contract test, not a silently widened audit log. If the agent grows a field, I want the same failure, on my laptop, before I open a pull request.

If you rerun this loop, keep the spec file as the only source of truth and treat the free server as a scratch checkout, not as production. That is the one setup I would copy.

Decision table I wish I had at hour zero

Situation	Do this	Do not do this
You have a captured body and no SDK	Write additionalProperties: false and generate the dataclass from that schema	Ask the agent to “fill typical fields”
You have an official OpenAPI file	Feed that file, then generate	Paste a route name and hope
Tests only exercise mocks the agent wrote	Replace one mock with a captured fixture	Add more mocks
Client dump writes keys the schema omits	Fail the build	Default the keys to None or 0.0
Regenerating on another machine	Copy contracts/ and fixtures/ first	Copy the previous generated module
Secrets, tokens, live customer payloads	Keep them off the scratch box	Paste real Authorization headers into the prompt

What I would repeat

I would repeat the boring parts. I would capture one fixture before I ask for any client. I would put additionalProperties: false in the schema on purpose, because that flag is the entire conversation with the agent. I would run Draft202012Validator.check_schema so a broken spec cannot hide behind a broken client. I would keep invented-key tests as parametrized cases, because those names are the residue of the first prompt.

I would also repeat the prompt shape, almost verbatim:

Read contracts/billing_event.schema.json.
Generate a dataclass with exactly those properties.
Do not add fields from common billing APIs.
Do not write mocks. I will supply fixtures/evt_9f3a.json.
If a field is not in the schema, it does not exist.

Would that prompt have saved me forty-eight hours? Probably not all of them. It would have saved the twelve hours I spent tuning retries for a contract bug.

Who should not use this, and what it will not do

Skip this loop if you already generate clients from a published OpenAPI document with a real codegen pipeline. Skip it if you cannot capture a single honest fixture, because then you are still testing the agent’s imagination. Skip it if the payload is streaming, binary, or signed in a way JSON Schema cannot see. This is a shape check, not an authorization check, not a pagination check, and not a load test.

Limitations I hit while writing the notes:

• JSON Schema will not tell you that status: posted is business-wrong for a voided invoice.
• format: date-time depends on the validator and the extra format checker you actually enable.
• additionalProperties: false is hostile to additive API evolution; you need a versioned contract or a deliberate unevaluatedProperties policy.
• A free scratch server does not make a fixture authentic. Garbage in still compiles.
• First-person debugging notes are not a substitute for the vendor’s real SDK, if one exists.

I am still asking myself the same question I should have asked at hour one. If the agent can invent a field, why is my test suite willing to believe it? Pin the spec, capture one body, and let the dataclass stay smaller than the prompt that created it.