Postmortem: An HTML Interstitial Certified an Empty Diff Summary

Conclusion first: a release gate checked HTTP status and body truthiness. An HTML login page returned with 200 OK therefore passed as a valid model summary. The gate stayed green while the "summary" was a captive-portal screen.

This postmortem covers the timeline, the contributing factors, and the durable fix. It is a failure analysis, not a recap.

Scope note: the timeline below was reconstructed from a local reproduction harness, not from a customer incident. Every command and snippet is runnable as written.

What the gate actually checked

The gate had three conditions, and all three were satisfied by the wrong payload.

1. response.status_code == 200
2. len(response.text) > 0
3. no exception raised during the request

An HTML error page satisfies all three. The wrapper's JSON parse lived in a try/except that logged a warning and continued.

The design intent was reasonable. The implementation trusted transport metadata as a proxy for payload semantics. Those are different contracts.

Reconstructed timeline

All timestamps are UTC and come from the harness logs.

1. 09:12:04 — The nightly job posts a diff to the model endpoint from the free server instance.
2. 09:12:39 — A gateway in front of the endpoint answers with 200 and Content-Type: text/html, returning a sign-in interstitial.
3. 09:12:39 — The client logs status=200 bytes=412 and moves on. Bytes are non-zero, so nothing looks wrong.
4. 09:12:41 — The JSON decode fails. The wrapper catches ValueError, emits warn: non-JSON response, and returns "".
5. 09:12:42 — The gate reads the empty summary. For an empty string, the findings list is empty. No findings means no blockers.
6. 09:12:43 — The patch is labeled verified and queued for merge.
7. 09:52:10 — A human notices the summary pane is blank and the run is aborted.

Twenty bytes of Content-Type was the only honest signal in the whole exchange. Nobody read it.

The reproduction harness

Four files reproduce the failure end to end. Run them before trusting any summary pipeline.

1. A stub endpoint that returns HTML with a 200

# stub_endpoint.py
from http.server import BaseHTTPRequestHandler, HTTPServer

BODY = b"""<html><head><title>Sign in to continue</title></head>
<body>Session expired. Reload the page.</body></html>"""

class Stub(BaseHTTPRequestHandler):
    def do_POST(self):
        self.send_response(200)                         # status looks healthy
        self.send_header("Content-Type", "text/html")   # the only honest signal
        self.send_header("Content-Length", str(len(BODY)))
        self.end_headers()
        self.wfile.write(BODY)

    def log_message(self, *args):
        pass

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8099), Stub).serve_forever()

Start it with python stub_endpoint.py. The server answers every POST with the interstitial.

2. A contract validator that rejects it

The validator runs at the transport boundary. It rejects on shape before any downstream code sees the payload.

# contract.py
REQUIRED = {"summary": str, "findings": list}

class ContractError(RuntimeError):
    pass

def validate_summary(resp) -> dict:
    ctype = resp.headers.get("Content-Type", "").split(";")[0].strip().lower()
    if ctype != "application/json":
        raise ContractError(f"unexpected content-type: {ctype!r}")
    if resp.status_code != 200:
        raise ContractError(f"unexpected status: {resp.status_code}")
    try:
        payload = resp.json()
    except ValueError as exc:
        raise ContractError("body is not JSON") from exc
    if not isinstance(payload, dict):
        raise ContractError(f"top-level type: {type(payload).__name__}")
    for key, typ in REQUIRED.items():
        if key not in payload:
            raise ContractError(f"missing key: {key}")
        if not isinstance(payload[key], typ):
            raise ContractError(f"bad type for {key}")
    return payload

Note the .split(";")[0]. Real endpoints send application/json; charset=utf-8. A naive equality check on the raw header breaks valid responses.

3. A test that pins the failure

# test_contract.py
import requests, pytest
from contract import validate_summary, ContractError

ENDPOINT = "http://127.0.0.1:8099/"

def test_html_200_is_rejected():
    resp = requests.post(ENDPOINT, json={"diff": "..."})
    assert resp.status_code == 200          # transport says fine
    with pytest.raises(ContractError):      # contract says no
        validate_summary(resp)

Run pytest -q test_contract.py. The test fails against the old wrapper and passes against the validator. That difference is the whole fix.

4. A drift canary for the daily job

Contracts change without notice. A canary catches drift before a merge window opens.

#!/usr/bin/env bash
# shape-canary.sh
set -euo pipefail

curl -sS -D /tmp/h -o /tmp/b -X POST "$MODEL_ENDPOINT" \
  -H 'Content-Type: application/json' \
  -d '{"input":"ping"}'

grep -qi '^content-type: application/json' /tmp/h \
  || { echo "CONTRACT DRIFT: content-type"; exit 1; }

python -c 'import json,sys; json.load(open("/tmp/b"))' \
  || { echo "CONTRACT DRIFT: body"; exit 1; }

echo "canary ok"

Schedule it hourly. A canary failure should page, not warn.

Contributing factors

Five factors combined. None of them is a single root cause, which is why the fix is layered.

• Truthiness used as validity. A non-empty body is not a valid body.
• Exception swallowing. except ValueError returned an empty string instead of propagating an error.
• Metadata trust. Status code and byte count were treated as semantic guarantees.
• No negative fixtures. The test suite had no HTML-200 case, so the hole was untested.
• Observability gap. Logs captured status and size, never content type or a body hash.

Why the free endpoint path amplified it

Free model access and the free server option made this cheap to run, and that is the point. Cheap runs mean more runs, more retries, and more chances to sit behind a shared gateway.

Shared gateways emit interstitials. Session expiry, rate screens, and proxy notices all return HTML with a 200. The failure is a deployment property, not a product flaw.

A paid-only pipeline hits the same wall. It just hits it less often, which makes it harder to find.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator states the free tier includes an initial allocation of 10 million tokens alongside a free server option; confirm current terms on the project page, because quotas and terms change.

The durable fix

Four changes, in priority order.

1. Validate at the boundary. One function owns response shape. Nothing downstream parses raw transport output.
2. Fail closed. A ContractError must abort the run, never return an empty default.
3. Separate empty from invalid. findings: [] is a valid result. A missing findings key is not. Track those as different exit codes.
4. Canary the contract. Run shape-canary.sh on a schedule and alert on drift.

Exit codes make the distinction operational:

# 0 = valid summary, 2 = contract violation, 3 = transport failure
python run_summary.py || case $? in
  2) echo "contract violation - block merge"; exit 2 ;;
  3) echo "transport failure - retry later"; exit 3 ;;
esac

Limitations and who should not copy this

This approach has real costs.

• Latency and maintenance. The validator adds a parse and a schema that someone must version.
• Streaming breaks the model. SSE responses need a streaming-aware terminator check, not a whole-body validation.
• Semantic errors survive. A perfectly shaped summary can still be wrong. Schema validation is not evaluation.
• Strict content-type checks reject quirky but valid servers. Test against your real endpoint before rollout.

Teams without an owner for the response schema should not adopt this. Single-endpoint prototypes under one developer should not either. The overhead exceeds the risk at that scale.

Verification checklist before you ship

• Post a diff and confirm the summary pane is non-empty and human-readable.
• Flip the stub to text/html and confirm the run blocks with exit code 2.
• Run the canary against the real endpoint and confirm it exits 0.
• Confirm logs record content type and a body hash on every call.

A green gate should mean the payload was understood. Status codes and byte counts cannot certify that. Only a validated contract can.

If you want to reproduce the harness, the free model access and free server option are enough to stand up both processes and watch the gate fail correctly.