Cut the Mock Server: A Weekend Contract Replay for Agent-Written Clients

An AI coding agent will invent response fields the HTTP API never returned. The weekend fix is not a mock server. It is a freeze file, a handful of recorded fixtures, and a replay harness that fails the client before those invented keys land in application code.

The workflow below is a reproducible template, not a production case study. Two files carry the contract. The generated client stays untrusted until replay exits 0. Everything else was cut.

The failure this project exists to prevent

Agentic coding tools now emit HTTP clients in minutes. They also guess. A missing property becomes an optional field. A nested envelope becomes a flat DTO. A 204 becomes a parsed JSON body. The patch compiles. The first real response does not.

Cheap generation makes that drift cheap to introduce and expensive to notice. A weekend is enough to freeze the wire shape and refuse extra keys. It is not enough to rebuild Pact, WireMock, or a company stub farm.

Scope that survived the cut

The working demo has four pieces:

1. A contract.freeze.json that lists path, method, status, and the exact keys allowed on the JSON object.
2. Fixtures under fixtures/ with request metadata and one recorded body.
3. A lint step that reads agent-written Python and rejects attribute access on forbidden keys.
4. A replay step that feeds the recorded body into the generated parser and asserts key equality.

The demo language is Python 3. The freeze format is language-agnostic. The harness does not start a TCP server and does not need a framework.

What got skipped on purpose

• No live network. Recorded fixtures only.
• No OpenAPI codegen pipeline. The freeze is hand-sized for one resource.
• No authentication refresh, pagination walkers, or webhook signatures.
• No CI matrix, Docker Compose, or service virtualization.
• No claim that a freeze replaces staging integration tests.

Those items turn a Saturday demo into a product. They stayed on the floor.

File 1: the freeze

A freeze is a JSON document the agent may read and the harness must enforce. Extra keys in generated models are defects, not features.

{
  "service": "billing-demo",
  "frozen_at": "2026-09-06",
  "resources": [
    {
      "id": "invoice_get",
      "method": "GET",
      "path": "/v1/invoices/{id}",
      "status": 200,
      "content_type": "application/json",
      "body_type": "object",
      "required_keys": ["id", "status", "amount_cents", "currency"],
      "optional_keys": ["paid_at"],
      "forbidden_keys": ["internal_ledger_id", "raw_gateway_payload"],
      "nested": {
        "amount_cents": "integer",
        "currency": "string",
        "status": "enum:draft|open|paid|void"
      }
    }
  ]
}

forbidden_keys matters. Agents copy names from comments, README fragments, and adjacent types. Listing keys that must never appear catches that copy without standing up a backend.

File 2: one recorded fixture

{
  "resource_id": "invoice_get",
  "request": {
    "method": "GET",
    "path": "/v1/invoices/inv_1001"
  },
  "response": {
    "status": 200,
    "headers": {"content-type": "application/json"},
    "body": {
      "id": "inv_1001",
      "status": "paid",
      "amount_cents": 4200,
      "currency": "USD",
      "paid_at": "2026-08-12T10:03:11Z"
    }
  }
}

The body is the source of truth for replay. If a later freeze adds a required key this fixture lacks, the harness fails closed. Update the fixture from a real capture, or drop the new key from the freeze. Do not let the agent invent a default.

The lint: generated source versus freeze keys

The script below is a conservative AST check. It does not execute the agent patch. It walks attribute names and string constants, then compares them to the freeze. Treat it as a gate, not as a type system.

#!/usr/bin/env python3
"""freeze_lint.py — reject agent-written keys outside contract.freeze.json"""
from __future__ import annotations

import ast
import json
import sys
from pathlib import Path

class KeyVisitor(ast.NodeVisitor):
    def __init__(self) -> None:
        self.names: set[str] = set()

    def visit_Attribute(self, node: ast.Attribute) -> None:
        self.names.add(node.attr)
        self.generic_visit(node)

    def visit_Constant(self, node: ast.Constant) -> None:
        if isinstance(node.value, str) and node.value.isidentifier():
            self.names.add(node.value)
        self.generic_visit(node)

def load_allowed(freeze_path: Path) -> tuple[set[str], set[str]]:
    data = json.loads(freeze_path.read_text())
    allowed: set[str] = set()
    forbidden: set[str] = set()
    for resource in data["resources"]:
        allowed.update(resource.get("required_keys", []))
        allowed.update(resource.get("optional_keys", []))
        forbidden.update(resource.get("forbidden_keys", []))
    return allowed, forbidden

def main() -> int:
    if len(sys.argv) != 3:
        print("usage: freeze_lint.py contract.freeze.json generated_client.py", file=sys.stderr)
        return 2
    allowed, forbidden = load_allowed(Path(sys.argv[1]))
    tree = ast.parse(Path(sys.argv[2]).read_text())
    visitor = KeyVisitor()
    visitor.visit(tree)
    noise = {
        "get", "post", "json", "headers", "raise_for_status",
        "Client", "Session", "loads", "dumps", "self", "cls",
    }
    hits = sorted((visitor.names & forbidden) - noise)
    extras = sorted((visitor.names - allowed - forbidden - noise))
    if hits:
        print("forbidden keys in generated client:")
        for name in hits:
            print(f"  - {name}")
    if extras:
        print("keys not listed in freeze (review):")
        for name in extras:
            print(f"  - {name}")
    return 1 if hits else 0

if __name__ == "__main__":
    raise SystemExit(main())

Forbidden keys fail the run. Unlisted keys print as review noise. That split keeps the weekend gate small. A later weekday can turn extras into failures.

status is both an HTTP idea and a resource field. The noise set is a weekend compromise and will hide some real hits. Read the review list once before ignoring it.

The replay: parser in, fixture body in, key set out

Replay does not need the generated client's network stack. It needs the function that turns a JSON object into a domain object. The demo assumes the agent was instructed to expose parse_invoice(payload: dict) -> dict and to return only freeze keys.

#!/usr/bin/env python3
"""replay_harness.py — execute parse_invoice against recorded fixtures."""
from __future__ import annotations

import importlib.util
import json
import sys
from pathlib import Path

def load_parser(path: Path):
    spec = importlib.util.spec_from_file_location("generated_client", path)
    mod = importlib.util.module_from_spec(spec)
    assert spec.loader is not None
    spec.loader.exec_module(mod)
    if not hasattr(mod, "parse_invoice"):
        raise SystemExit("generated client missing parse_invoice(payload)")
    return mod.parse_invoice

def freeze_key_set(resource: dict) -> set[str]:
    return set(resource.get("required_keys", [])) | set(resource.get("optional_keys", []))

def main() -> int:
    freeze = json.loads(Path("contract.freeze.json").read_text())
    resource = freeze["resources"][0]
    allowed = freeze_key_set(resource)
    required = set(resource["required_keys"])
    parse = load_parser(Path(sys.argv[1]))
    fixture = json.loads(Path("fixtures/invoice_get.json").read_text())
    body = fixture["response"]["body"]
    parsed = parse(body)
    if not isinstance(parsed, dict):
        print("parse_invoice must return a dict")
        return 1
    missing = sorted(required - parsed.keys())
    extra = sorted(set(parsed.keys()) - allowed)
    failed = False
    if missing:
        print("missing required keys:", ", ".join(missing))
        failed = True
    if extra:
        print("invented keys:", ", ".join(extra))
        failed = True
    if parsed.get("amount_cents") != body.get("amount_cents"):
        print("amount_cents mutated during parse")
        failed = True
    return 1 if failed else 0

if __name__ == "__main__":
    raise SystemExit(main())

A mutated integer is as bad as an invented key. Replay checks both. One happy-path body is enough for the weekend. It is not enough for production coverage.

Commands for the weekend loop

python3 -m venv .venv
source .venv/bin/activate

# 1. freeze first — do not prompt the agent without this file
cat contract.freeze.json

# 2. generate generated_client.py from the freeze only
# 3. lint, then replay
python3 freeze_lint.py contract.freeze.json generated_client.py
python3 replay_harness.py generated_client.py
echo $?

Prompt text belongs next to the freeze, not only in a chat transcript. A short, versioned prompt keeps the weekend reproducible:

Implement parse_invoice(payload: dict) -> dict in generated_client.py.
Return only keys listed in contract.freeze.json.
Do not add helpers that read forbidden_keys.
Do not call the network. Parsing only.

If lint fails, the next prompt is the lint output. If replay fails, the next prompt is the invented key list. The agent does not receive a new product requirement in that turn.

Decision table: freeze, skip, or record again

Symptom	Action this weekend	Do not do
Agent adds internal_ledger_id	Fail lint via forbidden_keys	Quietly strip the field in app code
Fixture lacks a new required key	Recapture one real response or drop the key	Let the agent default it to None
Parser returns an extra nested object	Fail replay	Write an adapter "just for now"
Status is 204 with an empty body	Add a second freeze resource	Reuse the 200 parser
Auth token in a fixture	Redact and recapture	Commit the fixture

The middle column is the scope cut. Adapters and defaults are how invented fields become permanent debt.

Where generation can run

The freeze file and the prompt are small. They can leave the laptop. The recorded fixture should be redacted first. A full repository does not need to travel with them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. When local quota is the constraint, MonkeyCode's free model access and free server option can run the generation step from those two files. Lint and replay still run on the laptop. The harness does not require that host, and it should keep working if the remote step is skipped.

Limitations

The AST lint is noisy on large clients. String constants that are not field names show up in the review list. Dynamic getattr and generated class hierarchies slip through. Replay only proves the parser against one happy-path body.

The freeze will rot. APIs add fields. A stale freeze rejects valid clients or, worse, stops being read. Keep the freeze next to the fixture in the same commit. If nobody owns that pair, do not start.

This approach is the wrong tool for binary payloads, streaming endpoints, GraphQL queries with unconstrained selections, and anything that needs transport-level replay such as TLS session behavior, cookies, or idempotency keys. Use an existing contract suite there.

Who should not use this

• Teams that already run Pact, schemathesis, Dredd, or equivalent on every PR.
• Anyone checking fixtures that still contain customer data, secrets, or internal ledger identifiers.
• Weekend builders trying to fully generate an SDK, including retries and pagination, in one sitting.
• Reviewers who will merge on compile-green without running replay_harness.py.

The demo is a stop-ship gate for invented JSON keys. It is not coverage.

What this weekend actually ships

Two JSON files, two Python scripts, and a prompt that forbids extra keys. That is the demo. The skipped mock server is a feature of the cut, not a backlog item for Sunday night.

Readers who already hold a freeze file and need a remote place to emit generated_client.py can use the free server option for that step. Keep replay local. Keep the fixture redacted. Merge only when both commands exit 0.