Replaying the Incident: Postmortems for LLM-Backed Features

Your page goes off at 02:14, and the alert says your summarizer endpoint has been answering with prose where your parser expects a JSON object. Nobody deployed a prompt change, nobody touched the parser, and the latency graph looks perfectly healthy, which is exactly why you spend the next two hours guessing. The incident below is an illustrative composite rather than a report from a specific company, so treat the timeline as a template you fill with your own telemetry, and keep your own numbers.

Most postmortems written in that state are recaps: a chronological story with a happy ending, in which the service recovered and everyone learned something vague about monitoring. A recap tells you what happened to you; it does not change what your next deploy is able to do. The honest test of a postmortem is whether someone who was not on call can replay it, and whether a test now fails when the durable fix is reverted.

A timeline assembled from memory is fiction with timestamps, so the first artifact worth building is mechanical rather than narrative. Any three of your tools already emit ordered events — the deploy pipeline, the alert manager, and whatever stores trace spans — and you can merge them into one file in about thirty lines of standard-library Python. Save this as timeline.py and point it at JSONL exports, each record carrying at least a timestamp and a label.

#!/usr/bin/env python3
"""Merge deploy, alert and trace records into one ordered incident timeline.

Each input file is JSONL with at least {"ts": ISO-8601, "label": str}.
Usage: python timeline.py deploys.jsonl alerts.jsonl spans.jsonl
"""
import json, sys
from datetime import datetime, timezone

def load(path, kind):
    with open(path, encoding="utf-8") as fh:
        for line in fh:
            line = line.strip()
            if not line:
                continue
            rec = json.loads(line)
            yield {
                "ts": datetime.fromisoformat(rec["ts"].replace("Z", "+00:00")),
                "kind": kind,
                "label": rec["label"],
                "detail": rec.get("detail", ""),
            }

def main(sources):
    events = sorted(
        (e for kind, path in sources for e in load(path, kind)),
        key=lambda e: e["ts"],
    )
    print("| time (UTC) | source | event | detail |")
    print("| --- | --- | --- | --- |")
    for e in events:
        t = e["ts"].astimezone(timezone.utc)
        print(f"| {t:%H:%M:%S} | {e['kind']} | {e['label']} | {e['detail']} |")

if __name__ == "__main__":
    if len(sys.argv) != 4:
        sys.exit("usage: timeline.py deploys.jsonl alerts.jsonl spans.jsonl")
    main([("deploy", sys.argv[1]), ("alert", sys.argv[2]), ("trace", sys.argv[3])])

Run it once and read the output before you write a single sentence of narrative, because adjacency is where memory lies to you.

python timeline.py deploys.jsonl alerts.jsonl spans.jsonl > incident-timeline.md

What usually falls out is a deploy at 01:58 followed by the first malformed response at 02:03, a five-minute gap that everyone in the review will swear was thirty seconds. In this composite case, the contributing factors are unglamorous and individually survivable. An upstream provider retired the model alias your config pinned to, so requests quietly fell through to a newer default with looser JSON habits; your only test asserted HTTP 200 on a single happy-path prompt; retries re-asked the same question and kept partial failures out of the error-budget dashboard; and the alert watched p95 latency, which the fast, broken responses never touched.

None of those factors is a root cause in the singular sense, and that is the point of writing them as factors rather than as a villain. You shipped a contract you never tested, you relied on an identifier someone else controls, and your detection watched the wrong signal. The durable fix follows from the factors directly, which is how you know the analysis was real rather than decorative.

The step most teams skip is replay, because replaying a frozen prompt corpus two hundred times for a week costs tokens and needs somewhere to run that is not production. This is the narrow, honest place where MonkeyCode fits into the workflow: it is described as an open-source project offering free model access and a free server option, and the operator-supplied details put the free tier at 10 million tokens — confirm the current terms, model list, and limits on the project's own page, because quotas and catalogs change faster than articles do. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The useful role here is a sandbox endpoint for replay and a separate box to run the harness, so your production keys never enter the loop.

The harness itself is deliberately boring: freeze the prompts, call the endpoint, and count contract violations instead of judging output quality by eye. Point MODEL_BASE_URL at whichever OpenAI-compatible endpoint you are replaying against, and read the exact base URL and model identifier from that provider's documentation rather than from a blog post.

#!/usr/bin/env python3
"""Replay a frozen prompt corpus and measure contract violations, not vibes."""
import json, os, sys, urllib.request

BASE = os.environ["MODEL_BASE_URL"]
KEY = os.environ["MODEL_API_KEY"]
MODEL = os.environ["MODEL_NAME"]
REQUIRED = {"title", "summary"}   # what your feature actually promised downstream

def call(prompt):
    payload = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        f"{BASE.rstrip('/')}/chat/completions",
        data=payload,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"},
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)["choices"][0]["message"]["content"]

def violates(content):
    try:
        doc = json.loads(content)
    except json.JSONDecodeError:
        return True
    return not REQUIRED <= set(doc)

def main(corpus_path):
    prompts = [json.loads(l)["prompt"]
               for l in open(corpus_path, encoding="utf-8") if l.strip()]
    bad = sum(violates(call(p)) for p in prompts)
    rate = bad / len(prompts)
    print(f"{bad}/{len(prompts)} responses violated the contract ({rate:.1%})")
    sys.exit(1 if rate > 0.01 else 0)

if __name__ == "__main__":
    main(sys.argv[1])

Run it against the corpus and against a recorded sample from before the incident, and you will learn whether the behavior change was the model or your own code.

MODEL_BASE_URL=... MODEL_API_KEY=... MODEL_NAME=... python replay.py corpus.jsonl

That command turns into a test you keep, because a postmortem with no new failing test is a diary entry. Keep the corpus in the repository and assert on the violation rate in CI, so an upstream change surfaces in a pull request instead of in a pager at 02:14.

# tests/test_output_contract.py
import json
from replay import call, violates

CORPUS = [json.loads(line)["prompt"] for line in open("tests/corpus.jsonl")]

def test_contract_holds_on_frozen_corpus():
    bad = [p for p in CORPUS if violates(call(p))]
    assert not bad, f"{len(bad)}/{len(CORPUS)} prompts broke the output contract"

You can hold the document itself to the same standard, which stops future postmortems from quietly degrading into recaps.

python - <<'PY'
import pathlib, sys
required = ["## Timeline", "## Contributing factors", "## Detection gap", "## Durable fix"]
doc = pathlib.Path("postmortems/2026-10-09-summarizer.md").read_text()
missing = [h for h in required if h not in doc]
sys.exit(f"postmortem is missing: {', '.join(missing)}" if missing else 0)
PY

The detection gap section is the one people omit, and it is where the next incident is hiding: in this case, nothing watched the parse-failure ratio, so the fix is a counter and an alert on that ratio rather than another latency threshold.

Some teams should not follow this route at all, and saying so is part of keeping the advice honest. If your prompts carry personal or regulated data, do not send a replay corpus to any shared or free resource, and build the harness inside your own boundary instead. If you cannot correlate logs with a request identifier, fix that before replay, because a corpus of prompts without trace IDs tells you a rate but not which user saw it. Treat free access as a convenience for investigation rather than a permanence guarantee, and never put it on the critical path of the fix itself. And if the incident was data loss, a bad migration, or an auth bypass, replaying prompts is simply the wrong tool.

What you want the next time the page fires at 02:14 is a timeline nobody wrote from memory, a corpus you can replay in one command, and a factor list that produced an actual test. If you want to try the replay step before committing budget to it, starting on a free tier is a reasonable way to find out how noisy your own contract really is.