When HTTP 200 Was the Outage

You get the page at 02:14, and the dashboard looks almost polite about the rising errors. Error rates sit at four percent, enough to wake you and still too small to look like a fire. You open the last merge, and the diff reads like a cleanup rather than a gamble. An assistant drafted the patch after you pasted a stack trace, and CI returned green within three minutes.

You tell yourself this will be a five-minute revert, then you notice the expected 400s are missing. Clients still receive 200, the worker stays up, and the event log never records the payment. The on-call rotation has a name for this shape of failure, even if the runbook does not. It is the incident that your tests loved and that production quietly refused to honor.

This write-up is a postmortem of that night, reconstructed as a pattern you can reuse. Treat the timestamps as a teaching timeline, not as a claim about one named company. The useful part is the durable fix you can check in, not the embarrassment of the revert. You will recognize the shape even if your webhook provider uses a different envelope.

Timeline

At 01:47 someone asked a coding assistant to stop the worker crashing on malformed webhooks. The prompt was honest and incomplete, which is how most two a.m. prompts get written. It said the process died on decode errors, customers retried, and the queue backed up. The generated patch caught a broad exception, logged a warning, and returned HTTP 200 anyway.

At 01:52 the unit tests were rewritten so they would match the new response contract. The fixture was a short ASCII body, and the assertion only checked for status 200. Nobody asserted that a valid payment still reached the ledger after a successful parse. CI finished at 01:55, and the small branch merged at 01:58 because it looked like hygiene.

At 02:14 the error-rate alerts stayed quiet while a business metric began to drift downward. Paid invoices appeared in the provider dashboard and then vanished from your own internal ledger. That gap is the whole incident, sitting in the space between HTTP health and product truth. The HTTP layer looked healthier than it had been all week, yet the product was not.

By 02:31 you had reverted the merge and watched the silent drop in payments stop. By 02:40 the provider retries landed cleanly, and the ledger started filling in order. The remaining work was not restoring traffic to customers after the revert went out. It was preventing the next assistant-authored fix from teaching your test suite to lie.

Contributing factors

The direct cause was a handler that acknowledged events it had not actually processed yet. Returning 200 after a parse failure is a product decision, and it needed a human owner. The assistant optimized for the symptom you typed, which was stop the crash and retries. Keeping the process alive felt like reliability, although it was only a quieter kind of outage.

The tests were accomplices rather than guards, since they pinned status and ignored side effects. They locked the new status code and never locked the write that actually pays the company. When you only assert the shape of the response, you are grading the costume, not the play. You asked a very fast intern to silence an alarm, then graded that intern on silence.

A second factor was payload realism, which sounds dull until a provider adds a byte-order mark. Production webhooks arrived as UTF-8 with a BOM from one vendor and as batches from another. The unit fixture was a single pretty-printed object copied from a README on a calm afternoon. The generated parser decoded the body as text and never inspected the Content-Type header at all.

A third factor was review theater on a short pull request written after a long day. The model had already explained the change in confident prose that felt finished and kind. You can feel that pull of fluency when the diff is small and the build is green. Fluent text is not evidence, and it is a texture that makes skipped assertions feel reasonable.

None of this requires a famous model, and none of it is unique to a single vendor. The same incident happens when you paste a stack trace into any chat and ship the first quiet patch. Tired engineers do not need a villain. They need an invariant that survives a fluent explanation. That invariant is simple to say and easy to skip when the pager is still ringing.

A durable fix you can run tomorrow

The durable fix is not a ban on assistants, because that advice fails the first night you are alone. The fix is to separate the roles so drafting, attacking, and reproducing cannot collapse into one step. One pass may draft a change, a second pass must attack it, and a third must reproduce it. You run that third pass on data that looks like production, not on the README fixture.

Start by writing the incident as a timeline of facts before anyone proposes a moral lesson. Record time, signal, action, and result, and keep guesses out of that section on purpose. If a sentence contains "we thought," move it into contributing factors where belief belongs. If a sentence contains "the handler returned 200," it can stay in the factual timeline.

Then freeze a reproduction that an isolated process can run without touching live customer traffic. You want two printed numbers from that process: the HTTP status and the ledger row count. If those numbers can diverge, your tests must watch both, or they will lie on command. A modest free server is enough here, because you are not load-testing the public internet.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that currently offers free model access and a free server option. That is the only reason it belongs in this workflow instead of remaining a footnote. You can run the same checks with any isolated box and any model you already trust. The product is relevant as a place to reproduce and critique, not as an on-call deputy. Check the project itself for whatever limits are current before you plan a process around them.

Here is a compact reproduction you can save as repro_webhook.py and run against a captured body. It is deliberately small so you can read every branch before you trust the invariant. Keep captured bodies in versioned fixtures, including the ugly ones your provider actually sends at night. The script refuses to catch a bare Exception, and it refuses to ack work it did not do.

import json
from pathlib import Path


class Ledger:
    def __init__(self):
        self.rows = []

    def record(self, event):
        if "invoice_id" not in event or "amount" not in event:
            raise ValueError("ledger refused incomplete event")
        self.rows.append(event)


def parse_body(raw: bytes, content_type: str) -> dict:
    if "application/json" not in content_type:
        raise ValueError(f"unsupported content-type: {content_type}")
    text = raw.decode("utf-8-sig")  # tolerate BOM from some providers
    payload = json.loads(text)
    if not isinstance(payload, dict):
        raise ValueError("webhook root must be an object")
    return payload


def handle(raw: bytes, content_type: str, ledger: Ledger) -> int:
    try:
        event = parse_body(raw, content_type)
        ledger.record(event)
        return 200
    except (ValueError, json.JSONDecodeError, UnicodeDecodeError):
        # 400 tells the provider to stop. 200 would lie.
        return 400


def main():
    raw = Path("captured_webhook.bin").read_bytes()
    content_type = Path("captured_headers.txt").read_text().strip()
    ledger = Ledger()
    status = handle(raw, content_type, ledger)
    print(f"status={status} ledger_rows={len(ledger.rows)}")
    if status == 200 and len(ledger.rows) != 1:
        raise SystemExit("handler acknowledged work it did not do")
    if status == 400 and len(ledger.rows) != 0:
        raise SystemExit("handler rejected a payload it still wrote")


if __name__ == "__main__":
    main()

You can seed the captured files with a single known payment and then rerun after each suspected fix. A command that belongs in the incident notes looks like the block below, and it should fail closed.

printf '%s' 'application/json; charset=utf-8' > captured_headers.txt
python3 - <<'PY'
import json
from pathlib import Path
Path("captured_webhook.bin").write_bytes(
    b"\xef\xbb\xbf" + json.dumps({"invoice_id": "inv_9", "amount": 1500}).encode()
)
PY
python3 repro_webhook.py
# expected: status=200 ledger_rows=1

python3 - <<'PY'
from pathlib import Path
Path("captured_webhook.bin").write_bytes(b"{not json")
PY
python3 repro_webhook.py
# expected: status=400 ledger_rows=0, and a nonzero exit if those two disagree

Notice what the script treats as a failed reproduction rather than as a sign of health. A 200 with zero ledger rows exits nonzero, which is the invariant the original tests forgot. You should store BOM prefixes, extra whitespace, and extra fields beside the pretty happy-path fixture. If a captured file cannot be redacted, do not upload it to any shared reproduction host.

Now add contract tests that would have failed at 01:52, before the merge reached main. Treat the examples as a pattern you can paste, not as a measured benchmark of any model. Three cases carry the postmortem: reject garbage, record a valid payment, and accept a BOM. Together they encode the night in a form CI can fail without waiting for a customer.

import json


def test_malformed_json_does_not_ack():
    ledger = Ledger()
    status = handle(b"{not json", "application/json", ledger)
    assert status == 400
    assert ledger.rows == []


def test_valid_payment_is_recorded():
    ledger = Ledger()
    body = json.dumps({"invoice_id": "inv_9", "amount": 1500}).encode()
    status = handle(body, "application/json; charset=utf-8", ledger)
    assert status == 200
    assert ledger.rows[0]["invoice_id"] == "inv_9"


def test_bom_prefixed_payload_still_records():
    ledger = Ledger()
    body = b"\xef\xbb\xbf" + json.dumps({"invoice_id": "inv_2", "amount": 80}).encode()
    status = handle(body, "application/json", ledger)
    assert status == 200
    assert len(ledger.rows) == 1

Those tests prevent the silent ack, protect the path that makes money, and admit messy production. You can add batch payloads later if your provider concatenates events into one request body. Do not let a generated test delete those cases because they look redundant next to the ASCII fixture. Redundancy is the point, since production is not a single pretty-printed object in a README.

Let the model prosecute the diff

After the reproduction is green, you can still use free model access as a second reader. The prompt should forbid patch generation so the model cannot complete the story you started. Ask only for ways the handler can acknowledge work it did not write to the ledger. Paste the diff, the tests, and one captured payload, then wait for the holes not the praise.

A practical prompt belongs in the runbook, not in a chat you invent while the pager rings. Keep the wording boring so nobody improves it into a request for a cleverer patch. Save it next to the reproduction script so the tired version of you does not have to invent discipline.

You are reviewing an incident diff. Do not propose a patch.
List concrete ways this handler can return 200 without writing a ledger row.
For each way, name the test that would catch it, or say "untested."
Challenge Content-Type handling, encoding, batch payloads, and exception scope.
If you cannot find a hole, say so in one sentence and stop.

Run that review on a free model endpoint if you have one, then do the same pass yourself. Agreement is not validation, and you are looking for the hole that both of you missed. If you use any shared assistant for that pass, keep production credentials out of the prompt. Isolation is the feature you need, not a confident paragraph that restates the diff in nicer words.

Limitations, and who should skip this

This workflow will not help if the incident is saturation, a consensus bug, or a vendor outage. A single captured payload cannot stand in for load, lock order, or clock skew across regions. It also fails when your ledger is a fire-and-forget queue with no queryable row to count. Without an observable side effect, you are back to grading costumes while the play goes wrong.

Do not use a free shared server for payloads that still contain live customer secrets or tokens. Redact, tokenize, or replay from staging, even when that slows the first reproduction by an hour. Do not let the attacking model see production credentials so it can be more helpful to you. Helpfulness is how the original bug got merged, dressed as resilience in a three-line comment.

If a policy forbids third-party model inference, skip the prosecutor pass and keep the contract tests. The invariant in the reproduction script is the part that survives a legal review of tools. If your on-call already has a mature chaos practice, treat this as a complement, not a replacement. Assistants complete the story you started, including the ending where the alarm goes quiet again.

When the next 02:14 arrives, you will still want a fast draft, and you should allow one. Let it draft, then run it on an isolated server with the ugliest payload you kept. Refuse any 200 that did not write a row, even when the explanation sounds operationally wise. If you want a place to try that isolated loop, the MonkeyCode open-source project is one option worth reading.