DEV Community

Morgan Xu
Morgan Xu

Posted on

Postmortem: A Cold Start Looked Like a Refusal, and the Patch Queue Stopped

#testing #ai #devops #postmortem

The short version

A gate that reviews AI-generated patches must separate three outcomes.

They are: the patch passed, the patch failed review, and nobody knows yet.

Our gate only had two. A 502 from a cold upstream became blocked.

That is a verdict. It is not a fact. The queue stopped for the wrong reason.

This post is a postmortem of that failure class. It covers the timeline, the contributing factors, the reproduction harness, and the durable fix.

All timestamps below come from a local reproduction harness. No production numbers are claimed.

Scope and method

The failure mode was rebuilt in a small sandbox. One fake reviewer server. One gate script. One queue of ten patches.

The harness is included in full. Anyone can rerun it and watch the same misclassification happen.

Timeline

The harness used a fake reviewer that sleeps, then returns 502 upstream connect error. The gate called it four seconds after boot.

T+ Event Observed state
0s Reviewer process starts, port bound Gate reports reviewer "up"
1s Queue drains, first patch sent HTTP 502 returned
1s Gate classifies response blocked, patch marked unsafe
2s Patches 2-10 sent with no backoff Nine more blocked
6s Reviewer finishes warming up No retries scheduled
40s Operator reads the board Ten false rejections
55s Manual rerun of all ten patches Ten pass

The last row is the tell. Nothing changed except time.

Contributing factors

  1. A health check that proved liveness, not readiness. The port answered TCP. The upstream did not answer anything. The gate treated those as the same signal.
  2. Error classification by status family, not by meaning. Any non-200 became a rejection. Infrastructure errors and model judgements shared one branch.
  3. An empty-body branch with the same bug. A blank body also became blocked. Two paths, one wrong default.
  4. No idempotency key. Retries were impossible to add safely, so nobody added them.
  5. A two-value verdict type. pass and blocked cannot express "retry me". The type forced the wrong default.

The fifth factor is the root cause. The first four are how it got through review.

The reproduction harness

Start with the fake reviewer. It fails the way a cold upstream fails.

# harness/fake_reviewer.py
from http.server import BaseHTTPRequestHandler, HTTPServer
import json, sys, time

MODE = sys.argv[1] if len(sys.argv) > 1 else "cold"

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers.get("content-length", 0))
        self.rfile.read(n)
        if MODE == "cold":
            time.sleep(2)
            self.send_response(502)
            self.end_headers()
            self.wfile.write(b"upstream connect error")
            return
        self.send_response(200)
        self.send_header("content-type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps({"verdict": "pass"}).encode())
    def log_message(self, *args):
        pass

HTTPServer(("127.0.0.1", 8081), Handler).serve_forever()
Enter fullscreen mode Exit fullscreen mode

Now the buggy gate. It is twelve lines and it looks reasonable.

# harness/gate_buggy.py
import json, sys, urllib.request

def naive_verdict(status: int, body: str) -> str:
    if status != 200:
        return "blocked"        # infrastructure became a judgement
    if not body.strip():
        return "blocked"        # the same mistake, second path
    return json.loads(body)["verdict"]

status, body = 502, "upstream connect error"
print(naive_verdict(status, body), file=sys.stderr)
Enter fullscreen mode Exit fullscreen mode

Run both. The gate says blocked. Rerun four seconds later and it says pass.

python harness/fake_reviewer.py cold &
sleep 1 && python harness/gate_buggy.py
sleep 6 && python harness/gate_buggy.py
Enter fullscreen mode Exit fullscreen mode

The durable fix

Three changes. Each one removes a way for the gate to guess.

1. A tri-state verdict type

unknown is a first-class outcome. It is not a synonym for failure.

from enum import Enum

class Verdict(str, Enum):
    PASS = "pass"
    FAIL = "fail"
    UNKNOWN = "unknown"

INFRA = {408, 425, 429, 500, 502, 503, 504}

def classify(status: int, body: str) -> Verdict:
    if status in INFRA:
        return Verdict.UNKNOWN
    if status in (401, 403):
        return Verdict.UNKNOWN      # our credential problem, not a review
    if status >= 400:
        return Verdict.FAIL         # the reviewer rejected the request itself
    try:
        payload = json.loads(body or "{}")
    except json.JSONDecodeError:
        return Verdict.UNKNOWN
    v = payload.get("verdict")
    return Verdict(v) if v in {"pass", "fail"} else Verdict.UNKNOWN
Enter fullscreen mode Exit fullscreen mode

2. Retries with an idempotency key

The key is derived from the patch and the model tag. Two retries of the same patch collapse into one recorded review.

import hashlib, random, time

def request_key(patch: str, model_tag: str) -> str:
    return hashlib.sha256(f"{model_tag}:{patch}".encode()).hexdigest()[:16]

def review(call, patch: str, model_tag: str, attempts: int = 3):
    key = request_key(patch, model_tag)
    last = Verdict.UNKNOWN
    for i in range(attempts):
        status, body = call(patch, idempotency_key=key)
        last = classify(status, body)
        if last is not Verdict.UNKNOWN:
            return last
        time.sleep((2 ** i) * 0.5 + random.random() * 0.3)
    return last
Enter fullscreen mode Exit fullscreen mode

3. A gate policy that never guesses

Unknown is not green. It is a hold, and a hold has an owner.

def gate(v: Verdict) -> str:
    return {Verdict.PASS: "green", Verdict.FAIL: "red"}.get(v, "hold")
Enter fullscreen mode Exit fullscreen mode
Verdict Gate colour Merge allowed Owner
pass green yes none
fail red no patch author
unknown hold no on-call, 15 min SLA

4. A readiness probe, not a liveness probe

Probe the review path with a cheap payload. Require a valid JSON verdict before draining.

curl -sS -o /tmp/warm.json -w '%{http_code} %{time_total}\n' \
  -X POST "$REVIEWER_URL/review" \
  -H 'content-type: application/json' \
  -d '{"patch":"warmup","warmup":true}'
python -c 'import json;print(json.load(open("/tmp/warm.json"))["verdict"])'
Enter fullscreen mode Exit fullscreen mode

Verification plan

Add these tests before shipping the fix. Each one maps to a contributing factor.

  1. 502 at t=1s returns unknown, never blocked.
  2. Empty body with status 200 returns unknown.
  3. Truncated JSON with status 200 returns unknown.
  4. 401 returns unknown and pages the operator.
  5. Three consecutive unknown results produce hold, not green.
  6. The same patch retried twice writes one review record, not two.
  7. Queue drain waits for one successful readiness probe.

A gate that fails these tests is still guessing.

Where free access fits, and what I used

This kind of harness is cheap to run. The reviewer tier only needs a model call. The gate runner only needs one small process.

For the sandbox, the reviewer used MonkeyCode's free model access. The gate runner sat on the free server option.

Both kept the experiment at zero cost while the failure mode was being chased.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest notes. Free tiers change, so verify current limits on the project page before you size a queue. And a free runner is still a runner. It can cold-start, and it can be evicted mid-request.

That is exactly why the tri-state fix matters more on free infrastructure, not less.

Limitations

  • The harness uses a synthetic failure. Real cold starts are less polite.
  • unknown adds operator load. Every hold needs a human with a deadline.
  • Idempotency keys help only if the reviewer honours them.
  • This does not make a weak reviewer correct. It only stops it from lying.

Who should not use this

Do not adopt the tri-state gate if any of these are true.

  • Nobody is on call to clear hold rows.
  • The pipeline treats any non-green as a hard stop with no owner.
  • Patches are low risk and volume is very high on a shared queue.

In those cases, a two-value gate with a manual rerun policy is the smaller evil.

Takeaways

Separate infrastructure failure from review outcome. Make unknown a real state. Give it an owner.

Then a cold start is a delay, not a verdict. And the queue keeps moving.

If you want to rerun this harness, start from the project's current docs and check the free-tier terms yourself before you commit a queue to them.

Top comments (0)