A 90-Minute Spike for Health Checks That Can Fail

AI health checks often return 200 during real outages. Process liveness is not the same as readiness. A 90-minute spike can kill the lying probe.

This method uses one hypothesis and one clock. The clock is ninety minutes, not a backlog. Ship an honest /ready, or kill it.

Hypothesis

Unconstrained coding agents skip checks in health routes. They return 200 because the process started. Load balancers then keep sending live traffic.

• Kill if /ready stays 2xx while a required store is down.
• Ship if /ready returns 503 on failure, then 200 after recovery.

No extra metrics. No framework swap. No service redesign.

Why the spike is 90 minutes

Generated handlers look finished in code review. They fail only when production storage dies. That is operational debt from cheap generated code.

Agentic workflows also assume happy paths by default. The hidden assumption is easy to miss. The server bound a port, so the service must be healthy.

A hard clock stops prompt cycling. Evidence beats taste in review threads.

Minute plan

1. Minutes 0-10: fixture, kill tests, baseline run.
2. Minutes 10-40: one constrained agent pass.
3. Minutes 40-70: re-run tests, classify fail-open catches.
4. Minutes 70-85: one repair pass, no new features.
5. Minutes 85-90: fill the ship-or-kill note.

Stop at 90 minutes. Logging refactors do not count as shipping.

Artifact: stub, naive server, kill tests

This fixture is a proposed spike. It is not a production service. Keep it in a throwaway directory.

Store stub

# dep.py
import os

class DependencyDown(Exception):
    """Raised when the backing store cannot be reached."""

def ping_store() -> None:
    if os.environ.get("STORE_DOWN") == "1":
        raise DependencyDown("store unreachable")
    return None

STORE_DOWN=1 simulates a dead dependency. No container runtime is required for round one.

Baseline handler, labeled anti-pattern

# app.py
import os
from http.server import BaseHTTPRequestHandler, HTTPServer

class Handler(BaseHTTPRequestHandler):
    def log_message(self, fmt, *args):
        return

    def do_GET(self):
        if self.path in ("/health", "/ready"):
            body = b'{"status":"ok"}'
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
            return
        self.send_response(404)
        self.end_headers()

def run():
    host = os.environ.get("HOST", "127.0.0.1")
    port = int(os.environ.get("PORT", "8080"))
    HTTPServer((host, port), Handler).serve_forever()

if __name__ == "__main__":
    run()

The baseline never imports dep. Both routes stay green. That identity is the bug under test.

Kill tests

# test_health.py
import os
import subprocess
import time
import unittest
from http.client import HTTPConnection

class ReadySpike(unittest.TestCase):
    PORT = 18080

    def start_app(self, store_down):
        env = os.environ.copy()
        env["STORE_DOWN"] = "1" if store_down else "0"
        env["PORT"] = str(self.PORT)
        env["PYTHONUNBUFFERED"] = "1"
        self.proc = subprocess.Popen(["python", "app.py"], env=env)
        time.sleep(0.5)

    def tearDown(self):
        proc = getattr(self, "proc", None)
        if proc is None:
            return
        proc.terminate()
        try:
            proc.wait(timeout=5)
        except subprocess.TimeoutExpired:
            proc.kill()

    def status(self, path):
        conn = HTTPConnection("127.0.0.1", self.PORT, timeout=2)
        conn.request("GET", path)
        resp = conn.getresponse()
        body = resp.read()
        conn.close()
        return resp.status, body

    def test_ready_is_503_when_store_down(self):
        self.start_app(store_down=True)
        code, _ = self.status("/ready")
        self.assertEqual(code, 503)

    def test_ready_is_200_when_store_up(self):
        self.start_app(store_down=False)
        code, _ = self.status("/ready")
        self.assertEqual(code, 200)

    def test_liveness_stays_200_when_store_down(self):
        self.start_app(store_down=True)
        code, _ = self.status("/health")
        self.assertEqual(code, 200)

if __name__ == "__main__":
    unittest.main()

Run the tests before any agent edit:

python -m unittest test_health.py -v

The baseline must fail test_ready_is_503_when_store_down. That red line is the spike start. Do not edit tests to match a green lie.

Single prompt, then hands off

Do not negotiate architecture in chat. Paste one task block. Then wait.

Edit app.py only.
GET /ready must call ping_store() from dep.py.
On success: HTTP 200 and {"status":"ok"}.
On DependencyDown: HTTP 503 and {"status":"store_down"}.
GET /health stays process liveness and returns 200.
Do not catch Exception.
Do not catch BaseException.
Do not add retries, threads, or new files.

Reject diffs that add frameworks. Reject silent except Exception: pass. Reject retries that hide store loss.

Clock and process hygiene

date +%s > /tmp/ready_spike_start
python -m unittest test_health.py -v | tee /tmp/ready_round1.txt
# constrained agent edit of app.py happens here
python -m unittest test_health.py -v | tee /tmp/ready_round2.txt
echo $(( $(date +%s) - $(cat /tmp/ready_spike_start) ))

Use a fresh Python process each round. Do not keep a REPL open. Imported handlers go stale after edits.

Wall clock over 5400 seconds is a kill. Overtime is not a ship. A second day is a new spike.

Curl check for the same evidence

Tests can fail from process wiring. Curl is a second witness. Keep both.

PORT=8080 STORE_DOWN=1 python app.py &
pid=$!
sleep 0.5
echo "ready=$(curl -s -o /tmp/ready_body -w '%{http_code}' http://127.0.0.1:8080/ready)"
cat /tmp/ready_body; echo
echo "health=$(curl -s -o /tmp/health_body -w '%{http_code}' http://127.0.0.1:8080/health)"
cat /tmp/health_body; echo
kill "$pid"

Expect 503 and store_down after a successful repair. Expect 200 on /health in the same window. Expect 200 and ok from /ready on the baseline.

Liveness versus readiness

Route	Question	Store down	Store up
/health	Is the process up?	200	200
/ready	Can it take traffic?	503	200

Liveness may stay green during store loss. Readiness may not stay green. Mixing those meanings is a kill condition.

Decision table

Evidence after two passes	Verdict	Action
/ready is 2xx when STORE_DOWN=1	Kill	Remove /ready from the balancer
except Exception still returns 200	Kill	Fail-open probe
New files or frameworks appeared	Kill	Scope break
/health and /ready are identical	Kill	Mixed probe
/ready is 503 down and 200 up	Ship	Keep tests in CI
Clock exceeds 90 minutes	Kill	Stop; retry another day

Ship requires every ship row. One kill row blocks merge. Partial greens do not count.

Fail-open pattern to reject

Agents often emit this shape:

try:
    ping_store()
except Exception:
    pass
self.send_response(200)

That catch converts outages into success. Treat it as a kill. The second pass must delete it.

Required shape after repair:

import os
from http.server import BaseHTTPRequestHandler, HTTPServer
from dep import ping_store, DependencyDown

class Handler(BaseHTTPRequestHandler):
    def log_message(self, fmt, *args):
        return

    def _json(self, code, payload):
        body = payload.encode("utf-8")
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self):
        if self.path == "/health":
            self._json(200, '{"status":"ok"}')
            return
        if self.path == "/ready":
            try:
                ping_store()
            except DependencyDown:
                self._json(503, '{"status":"store_down"}')
                return
            self._json(200, '{"status":"ok"}')
            return
        self.send_response(404)
        self.end_headers()

def run():
    host = os.environ.get("HOST", "127.0.0.1")
    port = int(os.environ.get("PORT", "8080"))
    HTTPServer((host, port), Handler).serve_forever()

if __name__ == "__main__":
    run()

Catch only DependencyDown. Do not catch Exception. Do not add a retry loop around ping_store().

Where a free model budget fits

A spike needs an isolated shell and a token ceiling. It does not need a product tour.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. The operator-supplied token ceiling is 10 million free tokens. Use that budget for one constrained pass plus one repair. Do not treat leftover tokens as a reason to widen scope.

Suggested use:

1. Copy dep.py, app.py, and test_health.py into a clean directory.
2. Run the kill tests before any model edit.
3. Apply the single prompt on the free server.
4. Re-run the same tests. Do not weaken assertions.
5. Write the ship-or-kill note. Stop.

Token spend is not quality. An always-200 probe is still a kill. The ceiling is a budget cap, not a measured spend for this article.

Limitations

This spike does not prove cluster readiness. It does not inject real packet loss. STORE_DOWN is a flag, not a network partition.

HTTP 503 is a convention. Some meshes expect 500 instead. Freeze the status code in the test first.

One stub is not three dependencies. Extra stores wait for a later spike. Do not start a service mesh inside the timer.

Agent output varies by prompt and context. This is a method, not a benchmark. No model names, latency numbers, or rankings are claimed.

Orchestrator probe YAML stays out of scope here. Map /health and /ready only after the tests pass.

Who should skip this spike

• Services with no balancer or orchestrator probe.
• Codebases that already contract-test readiness.
• Teams replacing SRE review with a timer.
• Workloads that need heavy GPU generation.
• Vendor scorecards. There is no leaderboard here.

Skip it if /ready already fails on dependency loss. Pick another failure mode. Do not rerun a green probe for content.

Ship-or-kill note

spike: ready-probe
clock_s:
round1: fail|pass
round2: fail|pass
fail_open_catch: yes|no
mixed_probe: yes|no
ship: yes|no
reason:

Fill it at minute 90. Attach it to the PR. Do not argue without the file.

Always-green probes hide incidents from the balancer. Ninety minutes is enough to expose them. Keep the tests. Drop the route if it still lies.