Case Study: Freeze Webhook Assumptions in Fixtures Before an Agent Writes Handlers

You should freeze webhook assumptions in versioned fixtures before any agent is allowed to generate handler code. Agents fill gaps with plausible headers, status codes, and secret names that your runtime never promised. A small contract gate turns that failure mode into a merge blocker you can run on a laptop or a free server. The rest of this case study is a reproducible walkthrough of that gate on a tiny inbound webhook service.

Background: a tiny webhook that looked easy

You need a receiver that accepts signed POSTs, maps a stable payload into an internal job, and returns a boring 202. Product wants a working receiver by Monday, so asking an agent to scaffold the handler looks like a reasonable shortcut. The first generated handler usually compiles, and that is exactly why you should not trust it yet.

The model tends to invent X-Signature-SHA256, WEBHOOK_SECRET, and payload.order_id because those names appear in training data. Your vendor actually sends X-Hook-Signature, reads HOOK_SHARED_KEY, and identifies the entity with payload.external_ref instead. Tests written by the same model assert the invented names, so they stay green while production stays dark.

This worked example treats that mismatch as the whole project, not as a later cleanup task. You will keep the service intentionally small so the gate stays readable. Nothing here claims a production incident, a customer count, or a latency number you cannot reproduce yourself.

Goal: make invented names unmergeable

You will treat the vendor contract as an input artifact, not as something the model is allowed to infer. The generator may write Python only after a schema, an assumption file, and a fixture payload already exist in git. The gate fails the patch when the handler mentions undeclared headers, env keys, or JSON paths. Success means a human can review handler logic instead of performing archaeology on invented header and field names.

The project boundary is deliberately narrow:

• One POST /hooks/vendor route that verifies a shared secret header.
• One JSON body whose allowed keys are listed in a schema file.
• One internal job record that copies declared fields and ignores the rest.
• Zero retries, zero queue workers, and zero vendor SDK wrappers in this pass.

If the agent wants a retry policy, a second route, or a new environment variable, it must update the frozen files first. That rule is the product of this case study, not a slogan around any particular coding assistant.

Frozen artifacts the agent is not allowed to invent

Create a directory that the generator can read but cannot silently expand. You should commit these files before the first prompt, then keep them in the same pull request as any later schema change.

webhook-receiver/
  contracts/
    assumptions.yml
    payload.schema.json
    accepted.json
  src/
    handler.py          # generated later, not first
  tests/
    test_contract.py
  tools/
    gate_assumptions.py
  Makefile

contracts/assumptions.yml is the human-readable allowlist. Keep every string literal that the handler may legally mention.

# contracts/assumptions.yml
route: POST /hooks/vendor
status_on_accept: 202
status_on_bad_signature: 401
status_on_invalid_json: 400
headers:
  signature: X-Hook-Signature
  content_type: Content-Type
env:
  - HOOK_SHARED_KEY
  - JOB_SINK_URL
payload_required:
  - external_ref
  - occurred_at
  - kind
payload_optional:
  - note
forbidden_substrings:
  - order_id
  - WEBHOOK_SECRET
  - X-Signature-SHA256

contracts/payload.schema.json is the machine-readable twin. You should reject extra properties so a generated parser cannot grow a shadow API.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": ["external_ref", "occurred_at", "kind"],
  "properties": {
    "external_ref": { "type": "string", "minLength": 1 },
    "occurred_at": { "type": "string", "format": "date-time" },
    "kind": { "type": "string", "enum": ["created", "updated", "canceled"] },
    "note": { "type": "string", "maxLength": 500 }
  }
}

contracts/accepted.json is one golden request body. The gate will parse the handler against assumptions, then run the handler against this fixture with a known signature header.

{
  "external_ref": "vnd_9f3a",
  "occurred_at": "2026-09-07T12:00:00Z",
  "kind": "updated",
  "note": "quantity changed"
}

Implementation: a gate that reads code, not vibes

The gate is ordinary Python. It loads the YAML allowlist, walks the handler AST, and fails when a string literal looks like a header, env key, or payload field that you never declared. This is a static check, so it does not need the vendor to be online.

# tools/gate_assumptions.py
from __future__ import annotations

import ast
import sys
from pathlib import Path

import yaml

ROOT = Path(__file__).resolve().parents[1]
HANDLER = ROOT / "src" / "handler.py"
ASSUMPTIONS = ROOT / "contracts" / "assumptions.yml"


def load_allowed() -> set[str]:
    data = yaml.safe_load(ASSUMPTIONS.read_text())
    allowed = {
        data["route"],
        data["headers"]["signature"],
        data["headers"]["content_type"],
        "/hooks/vendor",
    }
    allowed.update(data["env"])
    allowed.update(data["payload_required"])
    allowed.update(data["payload_optional"])
    allowed.update(str(data["status_on_accept"]))
    allowed.update(str(data["status_on_bad_signature"]))
    allowed.update(str(data["status_on_invalid_json"]))
    return allowed


class StringCollector(ast.NodeVisitor):
    def __init__(self) -> None:
        self.strings: list[str] = []

    def visit_Constant(self, node: ast.Constant) -> None:
        if isinstance(node.value, str):
            self.strings.append(node.value)
        self.generic_visit(node)


def main() -> int:
    if not HANDLER.exists():
        print("gate fail: src/handler.py is missing")
        return 1
    allowed = load_allowed()
    tree = ast.parse(HANDLER.read_text())
    collector = StringCollector()
    collector.visit(tree)
    data = yaml.safe_load(ASSUMPTIONS.read_text())
    forbidden = set(data["forbidden_substrings"])
    problems: list[str] = []
    for value in collector.strings:
        if value in forbidden:
            problems.append(f"forbidden literal {value!r}")
        looks_like_header = value.startswith("X-") or value.endswith("Signature")
        looks_like_env = value.isupper() and "_" in value
        looks_like_field = value in {"order_id", "user_id", "event_name"}
        if (looks_like_header or looks_like_env or looks_like_field) and value not in allowed:
            problems.append(f"undeclared literal {value!r}")
    if problems:
        print("gate fail:")
        for item in problems:
            print(f"  - {item}")
        return 1
    print("gate ok: handler literals stay inside contracts/assumptions.yml")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Pair the static gate with one runtime test that uses the golden fixture. The test should construct the signature from HOOK_SHARED_KEY so the agent cannot invent a second secret just to make hashing convenient.

# tests/test_contract.py
import hashlib
import hmac
import json
import os
from pathlib import Path

from jsonschema import Draft202012Validator

ROOT = Path(__file__).resolve().parents[1]
SCHEMA = json.loads((ROOT / "contracts" / "payload.schema.json").read_text())
BODY = json.loads((ROOT / "contracts" / "accepted.json").read_text())


def test_golden_payload_matches_schema():
    Draft202012Validator(SCHEMA).validate(BODY)


def test_handler_accepts_declared_signature(monkeypatch):
    from src.handler import handle

    secret = "test-shared-key"
    monkeypatch.setenv("HOOK_SHARED_KEY", secret)
    monkeypatch.setenv("JOB_SINK_URL", "https://sink.example.invalid/jobs")
    raw = json.dumps(BODY, separators=(",", ":")).encode()
    signature = hmac.new(secret.encode(), raw, hashlib.sha256).hexdigest()
    status, _ = handle(
        headers={
            "X-Hook-Signature": signature,
            "Content-Type": "application/json",
        },
        body=raw,
    )
    assert status == 202

Wire both checks so you can run them before reading the generated patch. A Makefile keeps the order obvious when you paste commands into an agent prompt.

.PHONY: gate test
gate:
    python tools/gate_assumptions.py
test: gate
    pytest -q tests/test_contract.py

Run the empty tree first so you know the failure shape:

python -m venv .venv
source .venv/bin/activate
pip install pyyaml jsonschema pytest
mkdir -p src
printf 'def handle(headers, body):\n    raise NotImplementedError\n' > src/handler.py
make test

You should see the runtime test fail on NotImplementedError while the gate stays quiet, because the stub contains no invented literals yet. That split is useful: missing behavior is a test failure, and invented names are a contract failure.

Where a free model and a free server actually help

You can run this generate-and-gate loop on a laptop, but a throwaway server keeps vendor secrets and local git state apart. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is enough to iterate a contract gate like this without standing up your own GPU box.

Give the model a prompt that forbids schema authorship. You want implementation inside a frozen envelope, not a second undocumented envelope.

Implement src/handler.py only.
Do not edit files under contracts/.
Do not add environment variables, headers, or payload keys.
Stop if make test fails and wait for a human.
Return 202 only after HMAC of the raw body matches X-Hook-Signature.

Then loop until the gate and the test agree:

make test
# if gate fail: restore src/handler.py from git and tighten the prompt
# if test fail: read the traceback; do not let the model edit assumptions.yml

The product mention stops here because the method does not depend on any one assistant. If you already have a model runner, you should keep it and still commit the same three contract files.

Results of the worked pass

The first generated handler in this walkthrough imported a fictional verify_stripe_signature helper and read os.environ["WEBHOOK_SECRET"]. The gate printed forbidden literal 'WEBHOOK_SECRET' and undeclared literal 'WEBHOOK_SECRET', so the patch never reached the HMAC test. That is the result you want: a cheap static failure instead of a green unit test that encodes the wrong vendor.

The second draft stayed inside the allowlist and computed HMAC-SHA256 over the raw body. make test passed on the golden fixture, and a human still had to read the sink POST because the gate does not prove you called JOB_SINK_URL correctly. Contract coverage is not behavioral coverage, and this case study should not pretend otherwise.

A third prompt tried to add order_id as an alias for external_ref. The schema rejected extra properties when the test loaded accepted.json against a secretly edited schema, and the forbidden-substring list caught the alias in handler source. You should treat that double hit as a feature: schema and source scans fail in different ways when the model rewrites the world to match its training data.

Limitations, and who should skip this

This gate is a string and schema fence, not a security review. It will not catch timing-unsafe HMAC compares unless you add that test yourself. It will not catch an allowed field used with the wrong meaning, such as stuffing a URL into note. It will not catch network retries, SSRF through JOB_SINK_URL, or log lines that dump the full body.

Skip this approach when any of the following is true:

• You do not yet know the real vendor headers and are hoping the model will discover them.
• The webhook carries payment or health data that needs a threat model, not a weekend AST walk.
• Your handler is generated into a language this script cannot parse.
• Nobody will read the remaining logic after make test turns green.

You should also skip it if the team will let the agent edit assumptions.yml in the same unattended loop. Once the model owns the allowlist, you are back to invented contracts with extra ceremony.

Lessons learned

Freeze names before you freeze code, because agents are fluent at completing missing APIs. Put the allowlist in files that tests import, so a prompt cannot privately renegotiate the vendor. Keep the static gate and the runtime test separate, because invented literals and missing HMAC checks are different bugs. Use a free server and free model access only as a loop runner, then judge the patch with make test rather than with the model's confidence.

If you try this on your own inbound hook, start with one route and one golden body. Expand the assumption file only when a human has seen a real vendor payload. That discipline is the entire case study; the assistant you wrap around it is optional.