The Probe Stayed Green for 48 Hours. The Pool Had Been Dead Since Boot.

The first 503 landed before coffee, and the status dashboard still painted every replica a comforting green. Kubernetes had not restarted a single pod, because the probe kept returning 200 like a polite liar. Have you ever trusted a /healthz path just because a tutorial dropped it into main.py? I did, and I spent the next forty-eight hours taking field notes instead of pretending a generated handler had finished the job.

The lie a 200 status can tell

A process that accepts HTTP is not the same thing as a process that can serve useful traffic. My readiness probe hit a handler that built a tiny JSON body and returned immediately, every time. The Postgres pool had failed during startup, which is a sentence I wish were only a joke. Why would a generated health route ever notice that failure? It never imported the pool, and it never tried a checkout.

I reproduced the lie with a laptop module rather than a cluster, because I wanted failing tests without waiting on kubelet timers. The notes below are that lab, not a dashboard screenshot with invented latency numbers. If you cannot run pytest locally, this walkthrough will not help you yet.

Hour 0–8: I asked for a health check, not a contract

I needed a tight draft-break-rewrite loop for the probe text itself, with failing assertions sitting in the same directory. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access on the free server option to generate probe variants, then I ran pytest on my laptop against those drafts. The assistant was a fast typist with opinions, not the owner of production traffic.

The first draft looked professional, which is exactly how these handlers seduce a tired reviewer. It returned {"status": "ok"} from a framework route and swallowed every exception it might someday raise. Does that look like engineering, or like a green badge glued onto a crash? I pasted the draft into a file and wrote the test the generator had skipped.

# generated_probe.py — labeled example, not production
from fastapi import FastAPI

app = FastAPI()

@app.get("/healthz")
def healthz():
    try:
        return {"status": "ok"}
    except Exception:
        return {"status": "ok"}

Yes, the except branch is unreachable in that snippet, and that is the whole field note. The model produced a handler that could not fail even after I wired a real dependency later. I asked a follow-up to check the database, and it added SELECT 1 inside the same try. Then it returned 200 on failure anyway. Who is the probe for, the kubelet or my feelings?

The contract I should have written first

A probe is a function with a timeout, not a slogan painted on a JSON object. I split liveness from readiness because Kubernetes already does, and mixing them is how a brief pool blip becomes a restart storm. Liveness means restart me if this process is wedged. Readiness means stop sending traffic if I cannot do useful work. Have you watched a liveness probe kill the only replica that was still warming its cache?

Kubernetes documents the three probe types in the liveness, readiness, and startup probe guide. I treated that page as the contract, not as optional cluster folklore. Here is the lab module I actually kept after throwing away the always-green draft.

# probe.py
from __future__ import annotations

from dataclasses import dataclass
from typing import Protocol


class Pool(Protocol):
    def ping(self, timeout_s: float) -> bool:
        """Return True only if a real checkout succeeded inside timeout_s."""

    def is_closed(self) -> bool:
        ...


@dataclass(frozen=True)
class ProbeResult:
    live: bool
    ready: bool
    reason: str
    http_status: int


def evaluate_probe(pool: Pool | None, *, timeout_s: float = 0.2) -> ProbeResult:
    # A listening process is live even when dependencies are down.
    if pool is None or pool.is_closed():
        return ProbeResult(
            live=True,
            ready=False,
            reason="pool_unavailable",
            http_status=503,
        )
    try:
        ok = pool.ping(timeout_s)
    except Exception as exc:
        return ProbeResult(
            live=True,
            ready=False,
            reason=f"ping_error:{type(exc).__name__}",
            http_status=503,
        )
    if not ok:
        return ProbeResult(
            live=True,
            ready=False,
            reason="ping_timeout_or_false",
            http_status=503,
        )
    return ProbeResult(
        live=True,
        ready=True,
        reason="ok",
        http_status=200,
    )

Notice what this function refuses to do, even when a draft tries to be helpful. It never converts an exception into success, and it never claims readiness because the interpreter is still running. It also never sets live=False for a down pool, because that would ask the kubelet to restart a process that is not wedged. Would you restart nginx because Postgres rebooted in the next rack?

Hour 8–24: the fake pool that made the tests honest

I did not stand up Postgres for this argument, because I needed deterministic failures instead of another flaky integration job. The fake pool below records whether ping was called, which is the assertion the generated handler could not survive. If your health check never increments that counter, you do not have a health check. You have a liveness selfie.

# fake_pool.py
from __future__ import annotations

import time


class FakePool:
    def __init__(
        self,
        *,
        closed: bool = False,
        ping_ok: bool = True,
        delay_s: float = 0.0,
        error: Exception | None = None,
    ) -> None:
        self.closed = closed
        self.ping_ok = ping_ok
        self.delay_s = delay_s
        self.error = error
        self.ping_calls = 0

    def is_closed(self) -> bool:
        return self.closed

    def ping(self, timeout_s: float) -> bool:
        self.ping_calls += 1
        if self.delay_s:
            time.sleep(self.delay_s)
            if self.delay_s > timeout_s:
                return False
        if self.error is not None:
            raise self.error
        return self.ping_ok

I keep repeating the counter because I failed that test with the first three generated drafts. Each draft returned pretty JSON. None of them touched the pool.

The pytest file that ended the argument

# test_probe.py
from fake_pool import FakePool
from probe import evaluate_probe


def test_listening_process_is_live_when_pool_is_gone() -> None:
    result = evaluate_probe(None)
    assert result.live is True
    assert result.ready is False
    assert result.http_status == 503
    assert result.reason == "pool_unavailable"


def test_closed_pool_must_fail_readiness() -> None:
    pool = FakePool(closed=True)
    result = evaluate_probe(pool)
    assert result.ready is False
    assert pool.ping_calls == 0


def test_ready_path_must_call_ping() -> None:
    pool = FakePool(ping_ok=True)
    result = evaluate_probe(pool)
    assert result.ready is True
    assert result.http_status == 200
    assert pool.ping_calls == 1


def test_ping_exception_does_not_become_200() -> None:
    pool = FakePool(error=RuntimeError("pool exhausted"))
    result = evaluate_probe(pool)
    assert result.http_status == 503
    assert "ping_error" in result.reason


def test_slow_ping_is_not_ready() -> None:
    pool = FakePool(delay_s=0.5)
    result = evaluate_probe(pool, timeout_s=0.05)
    assert result.ready is False
    assert result.live is True

Run the lab like this, from the directory that holds the three files:

python -m pytest test_probe.py -q

The slow-ping test is the one I would have skipped if I had only eyeballed the generated route. A probe that hangs longer than timeoutSeconds makes the kubelet guess. Do you want the kubelet guessing, or do you want your function returning 503 first?

Decision table I taped next to the handler

Signal	live	ready	HTTP	Where it belongs
Interpreter wedged or deadlock	false	false	500	liveness only
Process up, pool missing or closed	true	false	503	readiness, not liveness
Ping exceeds the probe timeout	true	false	503	readiness; keep live open
Ping raises	true	false	503	never catch into 200
Ping ok	true	true	200	share the function, not the threshold

I still see teams point both probes at /healthz and then wonder why a blip became a restart loop. Separate the URLs even if they call the same evaluator. Pass a flag, or read the path, but do not let a down database suicide the pod.

Kubernetes sketch, labeled as unexecuted

This YAML is a shape for readers who already own a cluster. It is not a claim that I rolled it out, and it is not tuned for your SLO.

# Copy only the shape. Fill timeouts from your pool, not from a model.
livenessProbe:
  httpGet:
    path: /livez
    port: 8000
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /readyz
    port: 8000
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 1
startupProbe:
  httpGet:
    path: /livez
    port: 8000
  periodSeconds: 5
  failureThreshold: 30

Startup probes exist because migrations and cache fills are not deadlocks. Without one, a slow boot looks like a failed liveness check. Did your generated manifest include startupProbe? Mine did not, until I asked for it by name.

What broke when I trusted the draft

• The handler returned 200 after except Exception, which hides pool exhaustion behind a smile.
• SELECT 1 ran on a borrowed connection that never returned to the pool under load.
• Liveness and readiness shared one URL, so a database blip became a restart loop.
• The ping had no timeout, so the probe hung until the kubelet timeout did the real work.
• Import-time pool creation failed, and the module still exported app like nothing happened.
• Generated tests asserted status_code == 200 and never inspected ping_calls.

That last bullet is the failure mode I care about more than the framework choice. Tests that only confirm the happy JSON are not tests. They are souvenirs.

What I would repeat in the next forty-eight hours

1. Write FakePool and the five assertions before any framework route exists.
2. Ask the assistant for variants, then throw away any draft that cannot fail closed.
3. Keep ping timeouts strictly smaller than the probe timeoutSeconds value.
4. Split /livez and /readyz even when the evaluator is a single function.
5. Log reason as a structured field, not as a string you only see in a hex dump.

The draft loop was useful when I treated it like a junior pair who types quickly. I pasted failing pytest output back and demanded a handler that satisfied ping_calls. I did not ask it to invent cluster capacity numbers, and you should not either.

Limitations, and who should not copy this lab

This lab does not prove your production database is healthy in any interesting sense. A ping can succeed while a critical table is locked, a replica is stale, or TLS certificates are twelve hours from expiry. Connection pools with lazy checkout will look fine until the first real request arrives. If you need multi-region failover, this function is a door alarm, not a control plane.

Do not use this approach if you cannot change probes in the target cluster. Do not point liveness at readiness logic just because a model emitted one convenient URL. Do not skip human review because the draft compiled and the JSON looked tidy. Free model access does not make the kubelet more forgiving, and a free server option does not replace load tests you have not run.

I also would not use a coding assistant as the source of timeout numbers. Those come from your SLO, your pool's wait budget, and the kubelet's timeoutSeconds. If those three disagree, the probe is theater.

Field notes I am keeping

The useful artifact was the fake pool, not the generated route with the pretty status key. The useful habit was failing closed, not adding more JSON fields that nobody pages on. Would I still draft with an assistant tomorrow, after this mess? Yes, with pytest in the same directory and a rule that 200 requires evidence of a checkout. If you want that same draft-and-test loop, MonkeyCode's free model access and free server option are how I iterated on the probe text before the tests stayed local.