Case Study: Freeze Outbound HTTP Policy Before an Agent Writes the Client

You should freeze outbound HTTP policy in a checked-in file before any agent writes a client. Agents fill gaps with retries, extra headers, and unbounded timeouts that your production path never approved. A small schema, a validator, and two failing tests catch those inventions before they merge. This case study walks a tiny supplier-lookup service from a frozen contract to rejection of a bad patch.

Background

You are adding a read-only lookup against a supplier catalog for live SKU availability checks. The service is small, and the temptation is to let an agent draft the entire HTTP client. That shortcut usually injects policy the rest of your stack never agreed to, especially around time and retries. You need a gate that fails the patch when the generated client disagrees with a human-owned YAML file.

The failure mode is boring and expensive at the same time for teams shipping generated clients. An agent often adds three retries with exponential backoff because popular HTTP tutorials do that. It may also set a sixty-second timeout or follow redirects across hosts without asking you. None of those choices belong in a latency-sensitive availability check without an explicit written contract.

Goal

You want a repository check that treats HTTP policy as data, not as comments in a prompt. The agent may implement the client, but it may not invent timeouts, retry counts, or extra headers. If the implementation drifts, continuous integration fails on the policy file rather than on production incidents.

Success looks like three concrete outcomes you can reproduce on a laptop without extra infrastructure:

1. A versioned http-policy.yml file that lists timeout, retries, allowed methods, and allowed headers.
2. A validator that reads the client module and compares literals against that file.
3. A pair of tests that fail a plausible agent patch and pass a policy-compliant client.

This is a worked example you can copy, not a claim about one production incident or a measured company rollout.

The frozen policy artifact

Keep the contract tiny so a reviewer can finish it in one sitting without a design review. You are not designing a service mesh or a full resilience framework in this repository. You are preventing an agent from guessing transport behavior while it writes a lookup helper. Unknown keys should fail, because models like adding circuit_breaker fields you never discussed.

# http-policy.yml
service: supplier_availability
protocol: https
allowed_methods:
  - GET
timeout_ms: 800
retries: 0
follow_redirects: false
max_response_bytes: 65536
allowed_request_headers:
  - Accept
  - User-Agent
  - X-Request-Id
denied_request_headers:
  - Authorization
  - Cookie
user_agent: inventory-lookup/1.0
idempotency_header: null

Pair the YAML with a JSON Schema so accidental edits fail before the agent session starts. You should reject extra properties, non-GET methods, and any retry value other than zero. The numeric bounds below are sample constraints for this lookup, not a universal SLA you should copy blindly.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "service",
    "protocol",
    "allowed_methods",
    "timeout_ms",
    "retries",
    "follow_redirects",
    "max_response_bytes",
    "allowed_request_headers",
    "denied_request_headers",
    "user_agent"
  ],
  "properties": {
    "service": { "type": "string", "pattern": "^[a-z_]+$" },
    "protocol": { "enum": ["https"] },
    "allowed_methods": {
      "type": "array",
      "items": { "enum": ["GET"] },
      "minItems": 1,
      "maxItems": 1
    },
    "timeout_ms": { "type": "integer", "minimum": 100, "maximum": 2000 },
    "retries": { "type": "integer", "minimum": 0, "maximum": 0 },
    "follow_redirects": { "type": "boolean", "const": false },
    "max_response_bytes": { "type": "integer", "maximum": 65536 },
    "allowed_request_headers": {
      "type": "array",
      "items": { "type": "string" }
    },
    "denied_request_headers": {
      "type": "array",
      "items": { "type": "string" }
    },
    "user_agent": { "type": "string" },
    "idempotency_header": { "type": ["string", "null"] }
  }
}

Set real timeouts from measured p99 latency of the supplier, then encode that number in YAML. Do not let the model pick timeout=60 because that value appears in generic HTTP snippets.

Implementation: parse the client, do not trust comments

Static comparison beats prompt reminders because comments are not executable and they rot across sessions. You can parse a small Python client with the ast module and look for timeout, headers, and retry helpers. The script below is a complete checker for a constrained client style that uses keyword arguments. It will not understand clever wrappers, which is a limitation you should keep visible.

Parsing calls with ast

# check_http_policy.py
from __future__ import annotations

import ast
import pathlib
import sys

import yaml

POLICY_PATH = pathlib.Path("http-policy.yml")
CLIENT_PATH = pathlib.Path("supplier_client.py")


class PolicyError(Exception):
    pass


def load_policy() -> dict:
    data = yaml.safe_load(POLICY_PATH.read_text())
    if data.get("retries") != 0:
        raise PolicyError("policy retries must stay at 0 for this service")
    return data


def literal_keywords(call: ast.Call) -> dict[str, object]:
    out: dict[str, object] = {}
    for kw in call.keywords:
        if kw.arg is None:
            continue
        if isinstance(kw.value, ast.Constant):
            out[kw.arg] = kw.value.value
        elif isinstance(kw.value, ast.Dict):
            headers = {}
            for k, v in zip(kw.value.keys, kw.value.values):
                if isinstance(k, ast.Constant) and isinstance(v, ast.Constant):
                    headers[str(k.value)] = v.value
            out[kw.arg] = headers
    return out


def check_client(tree: ast.AST, policy: dict) -> list[str]:
    errors: list[str] = []
    retry_names = {"retry", "with_retry", "tenacity", "backoff"}
    for node in ast.walk(tree):
        if isinstance(node, ast.Call) and isinstance(node.func, ast.Name):
            if node.func.id.lower() in retry_names:
                errors.append("client imports or calls a retry helper")
        if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute):
            if node.func.attr in {"get", "request", "urlopen"}:
                args = literal_keywords(node)
                timeout = args.get("timeout")
                if timeout is None:
                    errors.append("HTTP call is missing a timeout keyword")
                else:
                    timeout_ms = int(float(timeout) * 1000) if float(timeout) < 20 else int(timeout)
                    if timeout_ms != policy["timeout_ms"]:
                        errors.append(
                            f"timeout {timeout_ms}ms != policy {policy['timeout_ms']}ms"
                        )
                headers = args.get("headers") or {}
                for name in headers:
                    if name not in policy["allowed_request_headers"]:
                        errors.append(f"header {name} is not in the allow list")
                    if name in policy["denied_request_headers"]:
                        errors.append(f"header {name} is explicitly denied")
    return errors


def main() -> int:
    policy = load_policy()
    tree = ast.parse(CLIENT_PATH.read_text())
    problems = check_client(tree, policy)
    for item in problems:
        print(f"POLICY_FAIL: {item}")
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main())

A compliant client stays dull on purpose so the checker can see every keyword. You hard-code the timeout from the policy and you refuse to retry failed reads. SKU validation stays in your module so the agent cannot send arbitrary paths to the supplier host.

# supplier_client.py
from __future__ import annotations

import json
import urllib.request

TIMEOUT_S = 0.8
USER_AGENT = "inventory-lookup/1.0"
MAX_BYTES = 65536


def lookup_sku(base_url: str, sku: str, request_id: str) -> dict:
    if not sku.isalnum():
        raise ValueError("sku must be alphanumeric")
    req = urllib.request.Request(
        url=f"{base_url}/availability/{sku}",
        method="GET",
        headers={
            "Accept": "application/json",
            "User-Agent": USER_AGENT,
            "X-Request-Id": request_id,
        },
    )
    with urllib.request.urlopen(req, timeout=TIMEOUT_S) as resp:
        raw = resp.read(MAX_BYTES + 1)
    if len(raw) > MAX_BYTES:
        raise ValueError("response exceeded max_response_bytes")
    return json.loads(raw.decode("utf-8"))

urllib.request.urlopen takes timeout in seconds, so the checker converts that literal into milliseconds. If you later switch the client to httpx, you must extend the visitor instead of hoping the model notices the unit change.

Tests that fail an invented client

Do not review the agent patch by vibe when a fixture can encode the failure. Drop tests next to the checker: one golden client, and one snippet that looks like a typical generated retry wrapper. The invented example is labeled as a fixture, not as code you found in a live outage.

# test_http_policy.py
import ast
import pathlib
import textwrap

import check_http_policy as chk


def test_compliant_client_has_no_policy_errors():
    tree = ast.parse(pathlib.Path("supplier_client.py").read_text())
    policy = chk.load_policy()
    assert chk.check_client(tree, policy) == []


def test_retry_wrapper_is_rejected():
    invented = textwrap.dedent(
        """
        import time
        import urllib.request

        def lookup_sku(base_url, sku, request_id):
            last_error = None
            for _ in range(3):
                try:
                    req = urllib.request.Request(
                        url=f"{base_url}/availability/{sku}",
                        method="GET",
                        headers={"Authorization": "Bearer secret"},
                    )
                    return urllib.request.urlopen(req).read()
                except Exception as exc:
                    last_error = exc
                    time.sleep(0.2)
            raise last_error
        """
    )
    policy = chk.load_policy()
    errors = chk.check_client(ast.parse(invented), policy)
    assert any("timeout" in e for e in errors)
    assert any("Authorization" in e for e in errors)

Commands you run locally

Run the suite before you ask any model to touch supplier_client.py. After the agent returns a patch, run the same commands again and treat POLICY_FAIL as a merge blocker. Keep the policy file out of the writable path you give the agent, or the freeze is theater.

python -m pip install pyyaml pytest
python -m pytest test_http_policy.py -q
python check_http_policy.py

If you want schema validation on the YAML itself, add a separate step with a JSON Schema library you already trust. Do not ask the model to “just confirm the policy looks fine,” because that returns fluent agreement without checking additionalProperties.

Running the review loop on a free coding server

You can run this gate on a laptop, and you should, because the checker has no model in the loop. When you want a model to draft the client, keep the policy file read-only in the prompt. Paste the checker output back as the next message instead of arguing about style in prose.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option for this kind of tight loop. That is enough to iterate on this small lookup service without standing up your own GPU box. If this gate already matches how you review agent patches, you can run the same loop there.

A practical sequence looks like this, and you should keep it boring on purpose:

1. Commit http-policy.yml and the schema before the agent session starts.
2. Ask the model only to implement lookup_sku against that file.
3. Run pytest and check_http_policy.py on the returned tree.
4. If the checker prints POLICY_FAIL, send that stdout back and refuse to expand scope.

Do not ask the model to improve reliability as a vague side quest during the same session. That phrase is how retries and extra headers appear in otherwise tidy patches. Your policy already defined reliability as failing fast under 800ms with no retries and no extra headers.

Results

This is a worked example, not a production benchmark, and you should treat the outcomes as fixture results. The compliant client produces zero POLICY_FAIL lines when you run the checker against supplier_client.py. The invented retry wrapper fails on the missing timeout keyword and on the Authorization header. The checker does not prove JSON parsing, DNS failures, or TLS behavior, so you still need contract tests against a recorded supplier payload.

What you gain is a review handle that is cheaper than reading a long generated client line by line. A reviewer can open the YAML file, confirm retries stay at zero, and ignore stylistic churn in the Python module. That split matters when agents produce more lines than your team can read before the next standup. You are not measuring latency wins here; you are measuring whether invented transport policy can merge.

Limitations and who should skip this

The AST visitor is brittle on purpose, and you should assume it will miss disguised calls. It will miss timeouts hidden behind helper factories, config objects, or **kwargs unpacking. It will also miss retries implemented as raw loops unless you extend the walk beyond named helpers and missing timeouts. If your HTTP stack is generated from OpenAPI with shared middleware, freeze the middleware config instead of parsing call sites.

You should not use this approach when:

• Your client already lives behind a shared SDK that owns retries centrally.
• You need multi-origin failover that cannot be expressed as retries: 0.
• The agent is allowed to edit YAML and Python in the same patch, which defeats the freeze.
• You lack a recorded supplier response, so you would be testing only the policy file.
• Your language runtime makes literal extraction from source unreliable without a real compiler API.

Keep secrets out of the client module even when the denied header list looks thorough. That list is a tripwire for generated code, not a threat model for credential handling. If a generated client needs credentials, inject them from the environment in a wrapper you own, and never let the agent print them into fixtures or logs.

Lessons learned

Freeze policy as data before you invite a model to write transport code for a small service. Prompts do not survive the next session, while a YAML file and a checker keep failing the same way. You should fail closed on unknown headers and on any retry helper, because those are the assumptions agents reach for first when a tutorial mentions robustness.

Split ownership along that boundary and write it into the pull request template. Humans edit http-policy.yml. Models may edit supplier_client.py. Continuous integration owns the comparison and prints POLICY_FAIL on drift. If you keep those three roles mixed, you will review invented reliability theater instead of SKU lookup behavior.

When the supplier later needs a longer timeout, change the policy file in its own pull request first. Then re-run the checker and update the client in a follow-up change that cites the new number. That order keeps the contract readable, and it stops an agent from quietly moving your latency budget while it claims to implement the client.