Your Agent Will Invent Timeouts. Pin Them in a Contract Test First

When an agent writes an HTTP client, the silent bugs live in timeouts, retries, and idempotency headers you never approved. You should freeze those numbers in a checked-in policy file, then fail CI if generated code drifts. This case study walks a small billing adapter from an empty repo to a gate that rejects invented defaults. The rest of the article stays useful even if you never touch a hosted coding environment.

Background: a tiny billing adapter, not a platform rewrite

You are adding a payments adapter that charges a saved method through one upstream HTTP API. The public surface stays small: one create_charge call, a JSON body, and an Idempotency-Key header on every POST. You do not need a new framework, a mesh, or a generated SDK from a vendor portal. Agents still fill the remaining gaps with confident defaults that compile cleanly and look like careful engineering.

They pick a thirty second timeout, three retries, exponential backoff, and sometimes a second header name the upstream does not honor. Those choices look professional in review, then fail in production as double charges or hung workers. This walkthrough treats the adapter as a case study you can copy into a throwaway repository tonight. Every file below is a labeled worked example, not a production incident from a named company.

Goal: fail closed before the client exists

You want three properties in place before any model is allowed to write Python glue:

1. Retry count, timeout, backoff, and idempotency header names live in one YAML file.
2. The HTTP client reads those values at runtime and never declares a second set of constants.
3. A checker fails the patch when a literal timeout, retry, or synonym header appears beside the client.

If a generated line cannot be traced to http_client_policy.yaml, it should not invent a number. That is the whole requirement for this case study, and it is smaller than an architecture review. You are protecting side effects, not decorating a service catalog.

The frozen policy artifact

Create the policy before you open a chat window, and keep the document boring on purpose. Side-effectful transport behavior belongs here. Base URLs, secrets, and account identifiers do not.

# http_client_policy.yaml
# Worked example: freeze transport behavior for the billing adapter.
version: 1
upstream: payments-api
allowed_methods:
  - POST
timeout_ms: 2500
connect_timeout_ms: 500
max_retries: 1
retry_on_status:
  - 429
  - 503
backoff_ms: 200
backoff_jitter: false
idempotency:
  header: Idempotency-Key
  required_on:
    - POST
forbidden_headers:
  - X-Retry-Count
  - X-Idempotency-Key
max_request_bytes: 8192

Read the refusals as carefully as the numbers. You retry once, only on 429 and 503, with jitter turned off so CI stays deterministic. You also ban synonym headers because agents love to send both Idempotency-Key and X-Idempotency-Key for imagined compatibility. Duplicate keys are how a single charge becomes two captures when the upstream treats them as different requests.

Implementation: a checker the agent cannot charm

The checker is a small Python script you run locally and in CI. It loads the YAML, scans billing_client.py for timeout and retry literals, and rejects forbidden header strings. You should treat unmatched numbers near timeout, retry, or sleep as merge blockers, not as style nits from a linter.

# check_http_policy.py
# Worked example: reject patches that invent transport constants.
from __future__ import annotations

import re
import sys
from pathlib import Path

import yaml

POLICY_PATH = Path("http_client_policy.yaml")
CLIENT_PATH = Path("billing_client.py")

TIMEOUT_RE = re.compile(r"timeout(?:_ms)?\s*=\s*(\d+)", re.I)
RETRY_RE = re.compile(r"(?:max_)?retries?\s*=\s*(\d+)", re.I)
SLEEP_RE = re.compile(r"(?:sleep|backoff(?:_ms)?)\s*=\s*(\d+)", re.I)
HEADER_RE = re.compile(r"['\"](X-[A-Za-z0-9-]+|Idempotency-Key)['\"]")


def load_policy() -> dict:
    if not POLICY_PATH.exists():
        sys.exit("missing http_client_policy.yaml")
    return yaml.safe_load(POLICY_PATH.read_text())


def main() -> int:
    policy = load_policy()
    if not CLIENT_PATH.exists():
        print("billing_client.py is missing; refuse to invent one without the policy")
        return 2

    source = CLIENT_PATH.read_text(encoding="utf-8")
    timeouts = {int(m.group(1)) for m in TIMEOUT_RE.finditer(source)}
    retries = {int(m.group(1)) for m in RETRY_RE.finditer(source)}
    sleeps = {int(m.group(1)) for m in SLEEP_RE.finditer(source)}
    headers = {m.group(1) for m in HEADER_RE.finditer(source)}

    allowed_timeouts = {policy["timeout_ms"], policy["connect_timeout_ms"]}
    invented_timeouts = timeouts - allowed_timeouts
    if invented_timeouts:
        print("invented timeout literals:", sorted(invented_timeouts))
        return 1

    if retries and retries != {policy["max_retries"]}:
        print("invented retry literals:", sorted(retries))
        return 1

    allowed_sleeps = {policy["backoff_ms"]}
    if sleeps - allowed_sleeps:
        print("invented backoff literals:", sorted(sleeps - allowed_sleeps))
        return 1

    forbidden = set(policy["forbidden_headers"])
    if headers & forbidden:
        print("forbidden idempotency synonyms:", sorted(headers & forbidden))
        return 1

    required = policy["idempotency"]["header"]
    if required not in source:
        print(f"client never mentions {required}")
        return 1

    if "timeout_ms" not in source or "load_policy" not in source:
        print("client does not load the frozen policy at runtime")
        return 1

    print("http client policy holds")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The client must read the file, not shadow it

A client that copies 2500 into a constant still drifts the next time a model "cleans up" the module. Force the adapter to load YAML so the checker and the runtime share one source. That keeps a refactor from reintroducing seconds-based timeouts under a friendlier name.

# billing_client.py
# Worked example: transport constants come from the frozen policy only.
from __future__ import annotations

from pathlib import Path
from typing import Any

import yaml


def load_policy() -> dict[str, Any]:
    return yaml.safe_load(Path("http_client_policy.yaml").read_text())


def build_headers(idempotency_key: str) -> dict[str, str]:
    policy = load_policy()
    header = policy["idempotency"]["header"]
    return {header: idempotency_key}


def transport_options() -> dict[str, Any]:
    policy = load_policy()
    return {
        "timeout_ms": policy["timeout_ms"],
        "connect_timeout_ms": policy["connect_timeout_ms"],
        "max_retries": policy["max_retries"],
        "retry_on_status": list(policy["retry_on_status"]),
        "backoff_ms": policy["backoff_ms"],
        "backoff_jitter": policy["backoff_jitter"],
    }

Run the gate before you accept a patch, and run it again after every regenerate:

python -m pip install pyyaml pytest
python check_http_policy.py
pytest -q test_http_policy.py

Tests that encode the story, not the model

You want pytest to fail for the same reasons the checker fails. That duplication is intentional: the tests are the narrative you read in review, and the checker is the gate a model cannot talk past. Keep both, even when they assert the same 2500.

# test_http_policy.py
# Worked example: contract tests for the billing adapter transport policy.
from pathlib import Path

import yaml

from billing_client import build_headers, transport_options
from check_http_policy import main


def test_policy_file_is_the_only_source_of_timeouts():
    policy = yaml.safe_load(Path("http_client_policy.yaml").read_text())
    opts = transport_options()
    assert opts["timeout_ms"] == policy["timeout_ms"] == 2500
    assert opts["connect_timeout_ms"] == 500
    assert opts["max_retries"] == 1
    assert opts["retry_on_status"] == [429, 503]
    assert opts["backoff_jitter"] is False


def test_idempotency_header_uses_the_frozen_name():
    headers = build_headers("chg_123")
    assert headers == {"Idempotency-Key": "chg_123"}


def test_checker_passes_on_the_golden_client():
    assert main() == 0

Add one negative check when you review agent output. Copy the generated client aside, inject timeout=30, and confirm check_http_policy.py returns non-zero. You are not scoring a model. You are proving the gate still bites after a fluent explanation of why thirty seconds is "safer."

What the agent tries instead

In this worked example, a typical agent patch does four things you should expect and reject:

• It adds timeout=30 beside an HTTP constructor because most tutorials count seconds, not milliseconds.
• It retries on every 5xx, including 500, which turns a charge error into a second capture attempt.
• It sends both Idempotency-Key and X-Idempotency-Key, claiming compatibility with undocumented gateways.
• It introduces time.sleep(1) in an exception handler, which is a backoff policy smuggled through control flow.

Each of those is a contract miss, not a formatting miss. You reject the patch, point at http_client_policy.yaml, and ask for a diff whose literals only exist in that file. If the model argues that jitter is a best practice, you still fail the job, because the YAML already answered that question.

A decision table you can paste into review

Invented choice	Why it looks reasonable	Why this adapter rejects it
timeout=30	Tutorial clients use seconds	YAML pins 2500 ms so workers cannot hang a queue
max_retries=3	Feels resilient	Extra retries duplicate a captured charge
Retry on all 5xx	Blanket backoff looks thorough	500 after capture is not a signal to send the body again
X-Idempotency-Key	Synonym feels compatible	Upstream treats a second header as a second request identity
backoff_jitter=true	Production wisdom	CI and replayed jobs must stay deterministic

Where a remote workspace fits

You can run this entire loop on a laptop with pytest and a text editor. If you want the agent to edit the checkout on a remote box instead of beside your other processes, MonkeyCode offers free model access and a free server option you can point at this repository.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The remote workspace does not replace the YAML file, the checker, or the tests. It only hosts the edit cycle so agent processes stay off your desktop. Keep secrets out of that workspace, and keep the policy file in git so a fresh session cannot renegotiate timeouts in chat. If the hosted run cannot execute python check_http_policy.py, you do not have a safer workflow; you only have a farther laptop.

Results from the worked example

After the gate existed, the adapter could not be treated as done until transport behavior matched the YAML. The checker caught seconds-based timeouts, extra retry helpers, synonym headers, and smuggled sleep calls in the sample patches. Pytest then confirmed the client still read timeout_ms: 2500 and required Idempotency-Key on POST.

You should not read that as a benchmark, a latency study, or a claim about model ranking. It is a reproducible workflow: freeze the side-effectful numbers, scan for literals, refuse synonym headers, and only then let a model write glue. The useful result is a red X you can explain in review without arguing with a paragraph of generated rationale.

Limitations and who should skip this

This approach is deliberately narrow, and regex will lie to you in several ordinary codebases. It will not protect you if the agent talks to the network during planning, or if you paste production URLs into the prompt. Literal scans miss constants built by arithmetic, pulled from a database, hidden behind os.getenv, or buried in generated stubs.

Skip this pattern when:

• Your HTTP stack already comes from a vendor SDK whose retry policy you do not control.
• You need adaptive concurrency or client-side load balancing that cannot live in static YAML.
• The repository is a prototype with no payment side effects, so invented timeouts are cheap.
• You cannot run even one CI job that executes check_http_policy.py on the patched tree.

Do not treat the YAML file as a service mesh, a secrets manager, or an organization-wide standard. It is a contract for one adapter, one upstream, and one class of duplicate-charge bugs. If your real risk is authz or schema drift, freeze those surfaces separately instead of overloading this file.

Lessons you can reuse tomorrow

Pin the numbers that create duplicate side effects before you pin architecture diagrams or folder layouts. Timeouts, retries, jitter flags, and idempotency names are cheaper to freeze than they are to debug after a double capture. Put those values in a file the generated client must load, then fail the patch when new literals appear beside constructors.

Keep the checker boring, the tests duplicated on purpose, and the review conversation pointed at the YAML rather than at the model's tone. If you already keep contracts in git, run the same gate on a free remote workspace and confirm the checker still fails closed when a timeout shows up in seconds.