The Probe Returned 200. The Worker Had Never Bound a Port.

Have you ever watched a deploy go green while the only worker you cared about stayed completely silent? I did, across a messy forty-eight hours, and the healthcheck was the liar. This note is about an HTTP probe that returned 200 and a background process that never bound a port. I rebuilt the failure as a tiny local example so you can replay it without my original queue.

I am writing it as field notes so the next green square does not talk me out of looking. The lesson is not that probes are useless. The lesson is that a probe which cannot fail is just a mood.

Hour 0: A green square, a quiet queue

I had a small Python service with two moving parts that I treated as one. An HTTP process answered /health, and a worker was supposed to drain a local queue. I ran a smoke script after boot, saw 200, and closed the laptop like a person who had finished the job.

The worker was meant to refresh a heartbeat file every fifteen seconds. I did not open that file, because the probe had already told a story I wanted to believe. Have you noticed how a green check makes every other signal feel optional?

I told myself the HTTP process was the service. That sentence is how the rest of the night got expensive.

Hour 6: I debugged the broker, not the process

Nothing drained. The HTTP log stayed polite. There was no traceback, which felt worse than a crash. I asked the wrong first question: is the queue empty because producers failed, or because the broker lost the messages?

I restarted the web process. I tailed the access log. I diffed .env against .env.example and found nothing dramatic. Every curl still returned 200, so the silence kept looking like a data problem instead of a process problem.

curl -sS -D- http://127.0.0.1:8000/health
# HTTP/1.1 200 OK
# ok

That response is a fact about one handler. It is not a fact about the worker. I had to say that out loud before I could hear it. Would you keep restarting a web process that is already doing the only thing you asked it to prove?

The smoke script that hypnotized me

My early smoke script looked like a grown-up check and still measured the wrong process. It waited for a port, curled a path, and exited zero. That is a connectivity test, not a worker test, and I had named it smoke_health.sh anyway.

#!/bin/sh
set -eu
curl -sf --max-time 2 http://127.0.0.1:8000/health >/dev/null
echo "health: ok"

If a script cannot print health: missing worker, it should not be allowed to print health: ok. I know that now. I did not know it at hour six, because the script and I wanted the same ending.

Hour 14: I asked a model to just add a healthcheck

I wanted a second opinion, so I pasted the service layout into a free coding model and asked it to add a probe. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to draft candidate probes, and I ran the same service on the free server option so my laptop's working directory would not hide the lie. A throwaway shell is enough for this checklist; you do not need a cluster to watch /tmp disappear.

The model did what most of us do under time pressure. It wired GET /health to a constant ok on the process that was already listening. Why would it open a second socket when the first one already answered? I accepted that patch because it compiled, started, and made curl look busy.

# lying_probe.py — example of the handler I should have rejected
from http.server import BaseHTTPRequestHandler, HTTPServer

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"ok\n")
            return
        self.send_response(404)
        self.end_headers()

    def log_message(self, format, *args):
        return

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8000), Handler).serve_forever()

That file is honest about HTTP. It is silent about work. I copied it to the remote shell, started it, and watched the smoke script celebrate again. The model had not seen my process table. I had not shown it one.

Hour 22: /tmp made the worker look alive, then not

The worker I actually wanted wrote a heartbeat under /tmp, because that path is always writable, right? On my laptop the file sat there all afternoon like a loyal pet. On the free remote shell it vanished after a recycle, and the next boot looked like a brand new machine.

python worker.py &
stat /tmp/worker.heartbeat
# File: /tmp/worker.heartbeat

# later, after the machine recycled
stat /tmp/worker.heartbeat
# stat: cannot statx '/tmp/worker.heartbeat': No such file or directory

Was the worker dead, or had the disk forgotten it? Both, at different hours, which is a vicious combination. I spent a long stretch treating a missing file as proof of a crash, then treating a present file as proof of life. Neither reading was complete, because /tmp was not a contract.

Have you ever used “the file exists” as a synonym for “the process is healthy”? I had. Existence is not freshness, and freshness on ephemeral disk is not persistence. The remote shell taught that cheaper than a production incident would have.

Hour 30: The port was bound. The worker was not.

This is the command that ended the superstition. I stopped asking the probe what it felt, and I asked the kernel who owned the socket. The answer was rude in the useful way.

ss -lptn 'sport = :8000'
ps aux | grep -E 'lying_probe|worker.py' | grep -v grep

The HTTP process owned 8000. No worker.py was in the table. The probe had never been in a position to see that, because it did not look. How long would you keep curling a port that belongs to the wrong program?

I then checked the bind address, because localhost on a remote shell is not the address another namespace would use, and a model will happily curl itself.

ss -lptn | awk 'NR==1 || /8000/'
curl -sS --max-time 2 http://127.0.0.1:8000/health || echo "loopback only"

If the process binds 127.0.0.1, a probe from another namespace is not a healthcheck. It is a diary entry. I had been reading my own diary and calling it operations.

The artifact: a probe that can fail

I replaced the constant handler with a check that must observe the worker. The worker writes an ISO timestamp into a path I control, not /tmp. The HTTP process reads that file, parses the timestamp, and returns 503 when the stamp is missing or stale.

# worker.py — example heartbeat writer
from datetime import datetime, timezone
from pathlib import Path
import time

path = Path("./var/worker.heartbeat")
path.parent.mkdir(parents=True, exist_ok=True)

while True:
    path.write_text(datetime.now(timezone.utc).isoformat(), encoding="utf-8")
    time.sleep(15)

# honest_probe.py — runnable example
from datetime import datetime, timezone, timedelta
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path

HEARTBEAT = Path("./var/worker.heartbeat")
MAX_AGE = timedelta(seconds=30)

def worker_is_fresh(now=None):
    now = now or datetime.now(timezone.utc)
    if not HEARTBEAT.exists():
        return False, "missing heartbeat"
    raw = HEARTBEAT.read_text(encoding="utf-8").strip()
    try:
        stamped = datetime.fromisoformat(raw)
    except ValueError:
        return False, "unreadable heartbeat"
    if stamped.tzinfo is None:
        return False, "naive timestamp"
    if now - stamped > MAX_AGE:
        return False, "stale heartbeat"
    return True, "ok"

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/health":
            self.send_response(404)
            self.end_headers()
            return
        ok, reason = worker_is_fresh()
        code = 200 if ok else 503
        self.send_response(code)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(f"{reason}\n".encode("utf-8"))

    def log_message(self, format, *args):
        return

if __name__ == "__main__":
    Path("./var").mkdir(parents=True, exist_ok=True)
    HTTPServer(("127.0.0.1", 8000), Handler).serve_forever()

A test that fails the way the night should have failed

I wanted pytest to refuse the old story. If the heartbeat is missing, stale, or naive about timezones, the helper must return false. That is the whole product requirement for this probe.

# test_probe.py
from datetime import datetime, timezone, timedelta
import honest_probe as probe

def test_missing_heartbeat(tmp_path, monkeypatch):
    monkeypatch.setattr(probe, "HEARTBEAT", tmp_path / "worker.heartbeat")
    ok, reason = probe.worker_is_fresh()
    assert ok is False
    assert reason == "missing heartbeat"

def test_stale_heartbeat(tmp_path, monkeypatch):
    path = tmp_path / "worker.heartbeat"
    old = datetime.now(timezone.utc) - timedelta(seconds=45)
    path.write_text(old.isoformat(), encoding="utf-8")
    monkeypatch.setattr(probe, "HEARTBEAT", path)
    ok, reason = probe.worker_is_fresh()
    assert ok is False
    assert reason == "stale heartbeat"

def test_fresh_heartbeat(tmp_path, monkeypatch):
    path = tmp_path / "worker.heartbeat"
    path.write_text(datetime.now(timezone.utc).isoformat(), encoding="utf-8")
    monkeypatch.setattr(probe, "HEARTBEAT", path)
    ok, reason = probe.worker_is_fresh()
    assert ok is True

Run it like this, and do not collect a different file by accident:

mkdir -p var
pytest -q test_probe.py
python honest_probe.py &
curl -sS -D- http://127.0.0.1:8000/health
# HTTP/1.1 503 Service Unavailable
# missing heartbeat

That 503 is the first honest sentence the box said to me. After I started worker.py in the same working directory, the same curl returned 200 with ok. The difference was not prettier logs. The difference was a probe that had permission to fail.

Decision table I wish I had at hour 0

Claim you wanted	Command or check	What it still does not prove
HTTP process is up	curl /health on the real bind address	Worker loop is running
Worker is running	ps plus a fresh heartbeat in a durable path	The worker is draining the right queue
Heartbeat is meaningful	Timestamp is timezone-aware and younger than 30s	The write is not on ephemeral /tmp
Remote box matches laptop	Same command, same cwd, same interpreter	/tmp and home will survive a recycle
Model-generated probe is safe	A test that fails when the worker is absent	The model looked at your actual process table

I keep that table next to the smoke script now. If a row cannot fail, I do not let it vote. Would you let a check that only returns 200 participate in a ship decision? I would not, not after this.

What broke, in one list

• The probe and the worker were different processes, and I let one speak for both.
• A free model optimized for a passing curl, not for a failing one.
• /tmp on a disposable remote shell is not a database, and it is not a status page.
• Loopback success is not the same thing as a reachable bind address.
• A missing heartbeat after a recycle looked like a crash, then like a success, then like nothing.

Would I ask a model for a probe again? Yes, but I would paste this table first and refuse a handler that cannot return 503. The model is fast at wiring a route. It is not the process table, and it is not ss.

What I would repeat

1. Start the worker, then the probe, then curl, in that order, on the machine that will actually run them.
2. Put heartbeat files inside the repo's var/ directory, or another path you control, never /tmp.
3. Add a test that fails when the file is missing, stale, or naive about timezones.
4. Ask ss who owns the port before you ask a model why traffic is quiet.
5. Recycle the remote shell once on purpose, then run the same checks, because persistence is a feature you have to witness.

The disposable remote box was useful because it forgot /tmp and refused to flatter my laptop. That forgetfulness is the whole point of repeating the checklist somewhere that is not your home directory. If your only environment is a long-lived laptop, you will keep shipping laptop truths.

Limitations, and who should skip this

This workflow is a local honesty check, not a cluster scheduler. It will not replace Kubernetes probes, systemd watchdog units, or a real queue lag metric. If you already have a worker sidecar with a dedicated liveness file on persistent disk, you do not need my table.

Do not use a constant 200 handler in front of anything that spends money or deletes data. Do not store secrets in heartbeat files. Do not treat a free remote shell as production capacity, and do not treat a model's first probe as evidence. If your process must bind a public interface, this loopback example is incomplete on purpose.

I still like a green square. I just want it to be expensive to earn. The next time a probe answers before a worker binds a port, I want the 503 waiting in the test file, not in hour thirty of a quiet queue.