I Trusted the Generated Client for 48 Hours. extra=allow Hid the Schema Drift.

Have you ever merged a generated HTTP client because every test the model wrote came back green? I did, and then I spent forty-eight hours writing field notes on a payload that staging still rejected without drama. The assistant was not trying to trick anyone on purpose. It was optimizing for a happy path that never existed on the actual wire.

The orders API only needed a handful of states and a small create body, which made the job look almost insultingly easy. Generated pydantic models imported on the first try, and the sample tests asserted status_code == 200 against a mocked transport. Why would I stand up another server when the type checker already looked calm and the pytest summary was a solid wall of dots?

Because those tests never spoke the protocol at all. They spoke the in-process types, and that gap is what the next two days were about.

Field notes, hour zero through twelve

I started by pasting a trimmed OpenAPI fragment into a coding session and asking for a typed client. The first draft used extra="allow" on every model, which felt generous and modern until unknown fields started vanishing into production logs. I asked the model to add tests, and it added more mocks that never left the process, which still felt like progress if you only read the summary line.

Here is what I actually tried, in order, before I admitted the suite was theatrical.

1. Regenerated the client from the same fragment with stricter wording about closed enums.
2. Asked for pytest coverage of error paths, still against httpx.MockTransport.
3. Ran mypy and ruff until both tools went quiet on the generated tree.
4. Hit a shared staging host once, then blamed the gateway when the body looked almost right.

None of those steps failed loudly, and that silence is the whole problem. Quiet success is how schema drift hides in a green pipeline.

What actually broke

The server rejected status: "packed" because the live enum was packed_out, not the shorter word the model had guessed from English. Pydantic accepted the string because I had left the field typed as unconstrained str after a "simplify the models" pass. The generated test still passed, because the mock returned whatever JSON the test itself had stuffed into the fixture, which is a round trip that never leaves your imagination.

Would you have caught that from a green pytest summary alone? I would not have, and I did not. The summary only knew about the mock, and the mock was a very polite liar.

A second break showed up when the API made tracking_number required on shipped orders only. My client omitted the field and still looked healthy. extra="allow" on the response model swallowed extra keys coming back, so a read path looked fine in logs and the write path still violated the contract.

The generated shape that failed the double

This is the reduced client I kept in the notes as a warning, not as something I would ship. It compiles, it looks typed, and it still cannot see the protocol.

# bad_client.py — example of the generated shape that failed the double
from typing import Any
import json
import urllib.request
from pydantic import BaseModel, ConfigDict

class OrderIn(BaseModel):
    model_config = ConfigDict(extra="allow")
    sku: str
    quantity: int
    status: str  # unconstrained on purpose; this is the bug
    tracking_number: str | None = None

def create_order(base: str, **fields: Any) -> dict:
    payload = OrderIn(**fields).model_dump(exclude_none=True)
    req = urllib.request.Request(
        f"{base}/orders",
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=2) as resp:
        return json.loads(resp.read().decode("utf-8"))

If you only unit-test OrderIn(**fields), this file will keep looking finished. The bug is not syntax. The bug is that the model is willing to carry any string, and then willing to forget fields the server did not ask it to remember.

The artifact: a local contract double

I stopped asking the model to invent tests and wrote a tiny contract double instead. The double is not the real service, and I would not pretend otherwise. It is a deliberately picky HTTP server that encodes the decision table I should have started with, before any generated file existed.

# contract_double.py
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

ALLOWED_STATUS = {"pending", "packed_out", "shipped", "cancelled"}
REQUIRED_WHEN_SHIPPED = {"sku", "quantity", "status", "tracking_number"}
REQUIRED_DEFAULT = {"sku", "quantity", "status"}

class ContractHandler(BaseHTTPRequestHandler):
    def _read_json(self):
        length = int(self.headers.get("Content-Length", "0"))
        raw = self.rfile.read(length) if length else b"{}"
        return json.loads(raw.decode("utf-8") or "{}")

    def _send(self, code, payload):
        body = json.dumps(payload).encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_POST(self):
        if self.path != "/orders":
            return self._send(404, {"error": "unknown_path"})
        data = self._read_json()
        status = data.get("status")
        if status not in ALLOWED_STATUS:
            return self._send(422, {"error": "unknown_status", "got": status})
        required = REQUIRED_WHEN_SHIPPED if status == "shipped" else REQUIRED_DEFAULT
        missing = sorted(required - set(data))
        if missing:
            return self._send(422, {"error": "missing_fields", "fields": missing})
        extra = sorted(set(data) - required - {"note"})
        if extra:
            return self._send(422, {"error": "unexpected_fields", "fields": extra})
        kept = {key: data[key] for key in required}
        return self._send(201, {"id": "ord_1", **kept})

    def log_message(self, format, *args):
        return

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8089), ContractHandler).serve_forever()

Run it in one terminal and leave it running while the client tests execute against loopback. If the double is down, the suite should fail closed, not skip.

python contract_double.py

Decision table I should have written first

Before another prompt, I wrote the cases on paper, then turned them into pytest. The table is the artifact I would keep even if every generated file were deleted tomorrow morning.

| Case | status | extra fields | tracking_number | expect |
| pending happy path | pending | none | omitted | 201 |
| packed typo from English | packed | none | omitted | 422 unknown_status |
| shipped missing tracking | shipped | none | omitted | 422 missing_fields |
| unknown warehouse key | pending | warehouse | omitted | 422 unexpected_fields |
| shipped happy path | shipped | none | present | 201 |

# test_contract_client.py
import json
import urllib.error
import urllib.request
import pytest

BASE = "http://127.0.0.1:8089/orders"

def post(payload):
    req = urllib.request.Request(
        BASE,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urllib.request.urlopen(req, timeout=2) as resp:
            return resp.status, json.loads(resp.read().decode("utf-8"))
    except urllib.error.HTTPError as exc:
        return exc.code, json.loads(exc.read().decode("utf-8"))

@pytest.mark.parametrize(
    "payload, code, error",
    [
        ({"sku": "A", "quantity": 1, "status": "pending"}, 201, None),
        ({"sku": "A", "quantity": 1, "status": "packed"}, 422, "unknown_status"),
        ({"sku": "A", "quantity": 1, "status": "shipped"}, 422, "missing_fields"),
        ({"sku": "A", "quantity": 1, "status": "pending", "warehouse": "east"}, 422, "unexpected_fields"),
        ({"sku": "A", "quantity": 1, "status": "shipped", "tracking_number": "1Z"}, 201, None),
    ],
)
def test_order_contract(payload, code, error):
    got_code, body = post(payload)
    assert got_code == code
    if error:
        assert body["error"] == error

Point the generated client at 127.0.0.1:8089 and rerun the same cases through its public method, not through a mock. If the client coerces enums or drops fields, these tests fail in language you can read without guessing.

python -m pytest test_contract_client.py -q

Hours twelve through forty-eight

I regenerated the client against this double instead of against vibes, and the tone of the session changed immediately. The model still proposed extra="allow" twice, plus a status: str field that would have reopened the original hole. I rejected both patches because the double answered with 422, not with a shrug and a docstring. Would I have noticed without the table sitting next to the prompt? Probably not, because the prose explanation always sounded careful.

Commands I actually kept in the notes, because curl still tells the truth faster than a summary of mocks:

# prove the double is picky before blaming the client
curl -sS -D - http://127.0.0.1:8089/orders \
  -H 'Content-Type: application/json' \
  -d '{"sku":"A","quantity":1,"status":"packed"}'

# fail closed on extra keys the producer never documented
curl -sS http://127.0.0.1:8089/orders \
  -H 'Content-Type: application/json' \
  -d '{"sku":"A","quantity":1,"status":"pending","warehouse":"east"}'

The useful loop was boring on purpose. I changed one row in the table, watched the double, then asked the assistant for a client patch that made that row pass. I did not ask it to invent more coverage. I asked it to make a specific case pass without loosening the handler, which is a much smaller and much kinder request.

The fix that finally held used a closed set and forbade surprise keys on the way in.

from typing import Literal, Optional
from pydantic import BaseModel, ConfigDict

OrderStatus = Literal["pending", "packed_out", "shipped", "cancelled"]

class OrderIn(BaseModel):
    model_config = ConfigDict(extra="forbid")
    sku: str
    quantity: int
    status: OrderStatus
    tracking_number: Optional[str] = None

That is not clever. It is just honest about what the wire already knew.

Where a scratch coding environment actually fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode in this loop because I needed a scratch place to regenerate the client and run the double without mixing that experiment into a laptop already full of other Python versions. Free model access and a free server option were enough to keep the contract double and the pytest process running while I edited the table. I am not attaching a model name, a quota, a hardware story, or a benchmark, because those claims go stale and I did not run a bake-off.

The product did not find the enum drift. The decision table did. The models were useful for drafting the handler and the parametrized tests after the table existed. If you strip every product name out of this article, the harness still stands, and that is the only reason the notes are worth keeping.

If you want a disposable box for this exact double-plus-pytest loop, the free server option is a reasonable place to try the same files.

What I would repeat

• Write the decision table before the first prompt, even if it is only five rows on paper.
• Keep extra="forbid" on request models until a real producer proves it needs slack.
• Type enums as literals or Enum, never as unconstrained str, when the wire is a closed set.
• Point generated clients at a picky double, not at MockTransport, until the table is green.
• Ask the assistant to make one failing row pass, instead of asking it to add more tests.

Who should not use this approach

This double is not a substitute for staging, load tests, or a signed OpenAPI document from the owning team. Do not point it at production hosts, and do not feed it real customer payloads just to see what happens. If your organization already has a contract-testing pipeline with Pact or a similar tool, use that pipeline and skip my toy server entirely.

Skip a shared scratch environment if your payloads cannot leave your network, or if you need guaranteed hardware, uptime, or a specific accelerator. I did not verify those properties, and you should not assume them from a product mention. Teams that cannot tolerate a short-lived scratch server should run the same files on localhost and stop there.

Forty-eight hours was a long walk for five rows in a table. I would still walk it again, because a wall of green mocks is not a protocol, and it never was.