Pin the Cassette SHA Before the Agent Writes an HTTP Test

Do not merge an agent-written API test until it replays a locked HTTP cassette. A green unit job is not a contract. If that test can still open the network, you shipped a flake machine.

Agents are good at emitting clients, fixtures, and “happy path” calls. They are bad at leaving the network alone. Your job is to make live HTTP a merge-blocking event, then give the agent a narrow lane: client code plus tests that replay a hash-pinned cassette.

This is not a model bake-off. It is a green-to-merge path you can keep in the repo.

The failure you are actually debugging

An agent adds tests/test_billing_client.py. CI is green on the author’s laptop. Overnight the job fails: DNS, 429, a staging field rename, a clock skew in Expires.

You did not lose coverage. You lost determinism. The test was a live probe wearing a pytest filename.

Treat every agent HTTP test as untrusted until three files exist together:

1. A cassette (recorded request/response).
2. A SHA256 pin of that cassette in CI.
3. A flake budget that forbids retries and forbids the network.

No pin, no merge.

What this gate checks

You are not asking “did the agent write tests?” You are asking four narrower questions:

1. Does the new test import a replay helper instead of requests/fetch against a live host?
2. Is every cassette path under fixtures/http/ and listed in cassettes.lock?
3. Does sha256sum of each cassette match the lockfile on main?
4. Is flake-budget.yml set to retries: 0 and allow_network: false for this job?

If any answer is no, fail the required check. Do not “warn.” Warnings do not block merge.

Decision table

Diff touches	Cassette lock	Network in test	Merge
Client only, tests replay existing cassette	hash unchanged	blocked	allow
New endpoint + new cassette	hash added in same PR	blocked	allow after human review of cassette
Tests only, cassette missing	missing	n/a	fail
Cassette bytes change, lockfile not updated	mismatch	blocked	fail
Any https:// in tests/ outside fixtures	n/a	open	fail
Agent edits .github/workflows to skip the check	n/a	n/a	fail (keep this job required and name-pinned)

Print this table in the PR template. Agents follow checklists more reliably than prose.

1. Record the cassette against a stub, not production

Stand up a local stub. Do not record against a shared staging host. Staging is not a fixture. It moves.

# labeled example: local OpenAPI stub, not a production capture
python -m http.server 4010 --directory ./stubs/billing &
curl -sS http://127.0.0.1:4010/invoices/inv_1 > /tmp/inv.json

Keep the stub tiny. One resource, one error shape, one pagination cursor. If the cassette contains timestamps that change, strip them in a normalizer before you hash.

# labeled example: normalize before lock
import json, hashlib, pathlib, re

DROP = {"request_id", "generated_at", "trace_id"}

def normalize(raw: bytes) -> bytes:
    data = json.loads(raw)
    if isinstance(data, dict):
        for k in list(data):
            if k in DROP:
                data.pop(k)
    text = json.dumps(data, sort_keys=True, separators=(",", ":"))
    text = re.sub(r"\d{4}-\d{2}-\d{2}T[^\"]+", "<ts>", text)
    return text.encode()

def pin(path: pathlib.Path) -> str:
    digest = hashlib.sha256(normalize(path.read_bytes())).hexdigest()
    return f"{digest}  {path.as_posix()}"

Hash the normalized bytes, not the raw capture. Otherwise the agent “updates” the cassette every run and the lockfile becomes theater.

2. Lock the cassette in the same PR as the client

cassettes.lock is a text file. One line per fixture. Commit it with the client change. Split the PR if the agent also rewrites unrelated snapshots.

# cassettes.lock
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  fixtures/http/get_invoice_inv_1.json

The lockfile is the contract. The YAML workflow is only the referee.

# .github/workflows/agent-http-replay.yml
name: agent-http-replay
on:
  pull_request:
    paths:
      - "src/**"
      - "tests/**"
      - "fixtures/http/**"
      - "cassettes.lock"
      - "flake-budget.yml"
jobs:
  replay:
    runs-on: ubuntu-latest
    timeout-minutes: 8
    steps:
      - uses: actions/checkout@v4
      - name: pin-and-replay
        run: bash scripts/check_agent_http_tests.sh

Keep timeout-minutes low. Agent jobs that wander the network should die, not soak.

3. Put the checks in a script the hook and CI both run

Hooks that only live in GitHub will be skipped on a laptop. Duplicate the same script in pre-commit and in Actions.

#!/usr/bin/env bash
# scripts/check_agent_http_tests.sh
set -euo pipefail

fail() { echo "::error::$1"; exit 1; }

test -f cassettes.lock || fail "cassettes.lock missing"
test -f flake-budget.yml || fail "flake-budget.yml missing"

python - <<'PY'
import sys, yaml, pathlib
b = yaml.safe_load(pathlib.Path("flake-budget.yml").read_text())
job = b["jobs"]["agent-http-replay"]
assert job.get("retries") == 0, "retries must be 0"
assert job.get("allow_network") is False, "network must be false"
PY

# live hosts in tests/ are a hard fail
if grep -RInE 'https?://[^/]*' tests --include='*.py' --include='*.ts' | grep -v 'fixtures/http'; then
  fail "live URL in tests/; use fixtures/http cassettes"
fi

# every cassette on disk must be locked; every lock line must exist
while read -r hash path; do
  [[ -z "${hash:-}" || "$hash" =~ ^# ]] && continue
  test -f "$path" || fail "locked cassette missing: $path"
  actual=$(python -c "from scripts.normalize import pin; from pathlib import Path; print(pin(Path('$path')).split()[0])")
  test "$actual" = "$hash" || fail "cassette hash mismatch: $path"
done < cassettes.lock

# tests that name an HTTP client must import the replay helper
if git diff --name-only origin/main...HEAD | grep -E '^tests/.*client' >/dev/null; then
  grep -RIn 'replay_cassette\|vcr\.use_cassette\|nock' tests >/dev/null \
    || fail "HTTP client tests must call a replay helper"
fi

echo "agent-http-replay: ok"

Install it once:

chmod +x scripts/check_agent_http_tests.sh
cat > .git/hooks/pre-push <<'HOOK'
#!/usr/bin/env bash
set -euo pipefail
bash scripts/check_agent_http_tests.sh
HOOK
chmod +x .git/hooks/pre-push

If someone bypasses the hook with --no-verify, CI still fails. The hook is for speed. The required check is for truth.

4. Write the test so replay is the only I/O

Label this as a template, not a captured production run.

# tests/test_billing_client.py — labeled example
from pathlib import Path
from billing_client import BillingClient
from http_replay import replay_cassette

CASSETTE = Path("fixtures/http/get_invoice_inv_1.json")

def test_get_invoice_shape():
    with replay_cassette(CASSETTE, allow_network=False):
        inv = BillingClient("http://cassette.local").get_invoice("inv_1")
    assert inv["id"] == "inv_1"
    assert "amount_cents" in inv
    assert "request_id" not in inv  # stripped by normalizer; do not assert chatter

allow_network=False must be an argument the helper enforces, not a comment. If the helper cannot intercept the socket, fail the job in the helper, not in a linter after the fact.

5. Give the agent a write budget, then measure it

Agents pad tests. Cap what they may touch.

# flake-budget.yml
jobs:
  agent-http-replay:
    retries: 0
    allow_network: false
    max_new_cassettes: 3
    max_test_lines: 200
    timeout_seconds: 60

Count new cassette files and new test lines in the PR diff. If the agent dumps a hundred near-duplicate GET fixtures, fail. You wanted a contract, not a traffic dump.

new_cassettes=$(git diff --name-only --diff-filter=A origin/main...HEAD -- fixtures/http | wc -l)
test "$new_cassettes" -le 3 || { echo "too many new cassettes"; exit 1; }

Short cap. Hard fail. No “we will tidy later.”

Where a free local agent fits

You still need something to draft the client test. Keep paid cloud keys out of GitHub Secrets if you can. Run the agent next to the stub, on a machine you control, then let CI only replay.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you want that draft loop without stuffing a vendor key into Actions, MonkeyCode’s free model access and free server option are one way to generate the client test locally, then push only the cassette, lockfile, and replay test. CI should never need the model. CI needs the pin.

Do not ask the agent to “refresh cassettes” in GitHub-hosted runners. That reintroduces the network. Generation is a laptop (or free server) step. Merge is a replay step.

Green-to-merge path, in order

1. Human or stub records one cassette; normalizer strips chatter; SHA goes into cassettes.lock.
2. Pre-push runs scripts/check_agent_http_tests.sh.
3. Required check agent-http-replay runs the same script on the PR head.
4. Branch protection requires that exact check name. Renames fail closed.
5. Merge. Not before.

If you already split merge work from soak work, keep this job on the merge lane. Soak can probe staging later, off the critical path, with a different check name. Do not let soak results silently satisfy merge.

Limitations

This gate does not prove the backend is correct. It proves the client still understands a frozen conversation. If the real API changed and nobody updated the stub, you will merge a faithful client of a dead contract. Pair this with a scheduled contract job against a versioned schema, not against a live shared tenant.

Cassettes can leak secrets. Scan fixtures for tokens before you hash. If a cassette contains Authorization, fail. Redact, re-record, then lock.

Binary bodies, streaming endpoints, and websocket upgrades do not belong here. Use a different artifact for those.

Who should not use this

Skip this if your tests are already fully in-process with no HTTP. Skip it if you cannot keep a stub. Skip it if the agent is allowed to edit workflow files that define the required check — fix that permission first. Skip it if you need live canaries on every PR; those belong in soak, with an explicit budget, not in merge.

You want merge to be boring. Pin the cassette SHA. Then let the agent write the client test. Anything else is a live request with extra steps.