I Trusted /health for 48 Hours. The Worker Still Held the Lock.

I spent forty-eight hours staring at a green dashboard while real requests died in a worker thread. Staging never flipped red, and the deploy pipeline kept shipping because every probe returned 200 on time. Have you ever trusted a health check that never touched the lock your background jobs actually needed? That was the entire mess, and it started with a generated probe I did not read carefully enough.

What I thought was broken

I blamed the load balancer first, then DNS, then a slow dependency that had not changed in weeks. The process was still up, the port was still open, and curl from the same network returned HTTP 200 in milliseconds. Why would I question a probe that looked textbook-correct and came back instantly every single time?

The user-facing symptom was a timeout on a write path that used a shared worker lock. Reads still looked fine, which made the green probe even more convincing during that first long night. I kept chasing the network because the one URL I watched could not tell the truth.

Hour 0–8: the generated check I accepted

I had asked a coding assistant for a small Python service with a background worker and a health endpoint. The model produced a friendly /health handler that never acquired the same lock the worker used for writes. I pasted it, shipped it, and only later asked whether liveness and readiness were even different questions.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to iterate on the probe text without inventing a new service shape. I reproduced the hang on the free server option so my laptop was not the only clock in the story.

Here is the shape of what I first ran, labeled as a reconstruction rather than production source.

# reconstruction: naive probe + worker sharing one lock
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

lock = threading.Lock()
state = {"writes": 0}

def worker():
    while True:
        with lock:
            # stand-in for a stuck dependency inside the critical section
            time.sleep(30)
            state["writes"] += 1
        time.sleep(0.05)

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path in ("/health", "/ready"):
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"ok")
            return
        if self.path == "/write":
            with lock:
                state["writes"] += 1
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"wrote")
            return
        self.send_response(404)
        self.end_headers()

    def log_message(self, fmt, *args):
        return

Does that /health handler tell you the worker is alive and able to take writes? It tells you an HTTP thread can still accept a socket and write ok. That is a different claim, and I learned it the slow way after trusting the wrong URL.

Hour 8–24: commands that kept lying

I ran the usual comfort commands, and they all agreed the process was fine on paper. That was the trap, because those commands inspected the listener and never asked about the lock.

ss -ltnp | grep 8080
curl -sS -D- http://127.0.0.1:8080/health
curl -sS -o /dev/null -w "%{http_code} %{time_total}\n" http://127.0.0.1:8080/health
ps -o pid,stat,wchan,cmd -p "$(pgrep -f probe_lock_demo)"

Every line looked healthy: ss showed LISTEN, curl printed 200, and ps showed an ordinary Sl. Would you have torn the service down at that point, or would you have kept chasing the network?

The first honest signal was a second curl against /write that hung until the worker released the lock.

# this is closer to the request users were actually making
curl -sS -m 2 http://127.0.0.1:8080/write || echo "write timed out"

That two-second timeout failed while /health still returned 200 without a moment of thinking. Once I saw that split, the next twenty hours were about proving it on purpose instead of restarting at random.

ThreadingHTTPServer hid the lock

ThreadingHTTPServer was part of why I stayed confused for so long on night one. One thread could answer /health while another thread sat on lock.acquire() inside /write. A single-threaded server would have made /health hang too, and I might have found the bug before dinner. Have you noticed how extra threads can hide a lock by keeping the wrong URL fast?

Hour 24–36: reproduce it on a throwaway box

I wanted a machine that was not my laptop, because local sleep timings lie when a lid closes. A second process table and a boring network path were enough to make the hang boringly repeatable. I did not need special hardware for this; I needed a place where time.sleep meant what it said.

On that box I added a readiness path that contends for the lock with a short timeout. That is the whole design change, and it is small enough to read in one sitting.

def lock_ready(timeout=0.2):
    acquired = lock.acquire(timeout=timeout)
    if not acquired:
        return False
    try:
        return True
    finally:
        lock.release()

Wire /ready to lock_ready() and keep /health as process liveness only, nothing fancier. Then the probes finally disagree, which is the signal I wish I had on hour one.

curl -sS -o /dev/null -w "health %{http_code}\n" http://127.0.0.1:8080/health
curl -sS -o /dev/null -w "ready %{http_code}\n" http://127.0.0.1:8080/ready

When the worker holds the lock for thirty seconds, health stays 200 and ready goes 503. That split is the artifact I should have demanded before the first deploy ever shipped.

I also dumped frames when ready failed twice in a row, because a 503 without a stack is just another green-to-red flip.

import faulthandler
import sys

def dump_frames():
    faulthandler.dump_traceback(file=sys.stderr)
    for tid, frame in sys._current_frames().items():
        code = frame.f_code
        print(tid, code.co_filename, frame.f_lineno, code.co_name)

Call dump_frames() from the 503 branch during a reproduction, not from the 200 branch in production traffic. You want the waiter and the holder, not a stack for every successful probe.

The artifact: a lock-aware probe test

I do not want this to be a vibes post, so here is a reproducible check you can run without my service. Save it as test_lock_probes.py and run it with the stdlib unittest runner on any CPython.

"""Lock-aware readiness vs naive liveness. Run: python test_lock_probes.py"""
import threading
import unittest


class LockProbeTest(unittest.TestCase):
    def setUp(self):
        self.lock = threading.Lock()

    def naive_health(self):
        return True  # socket accepted, process alive

    def ready(self, timeout=0.2):
        got = self.lock.acquire(timeout=timeout)
        if not got:
            return False
        self.lock.release()
        return True

    def test_naive_health_stays_true_while_lock_is_held(self):
        self.lock.acquire()
        self.addCleanup(self.lock.release)
        self.assertTrue(self.naive_health())
        self.assertFalse(self.ready(timeout=0.05))

    def test_ready_recovers_after_release(self):
        self.lock.acquire()
        self.lock.release()
        self.assertTrue(self.ready(timeout=0.05))


if __name__ == "__main__":
    unittest.main()

If that first test ever fails on your machine, your mental model of "the server is up" is already safer than mine was. If it passes, you have a documented reason to split liveness from readiness before the next assistant-generated handler lands.

Decision table I now keep in the runbook

• TCP connect / port open: proves a listener exists. It does not prove the app can take work.
• HTTP /health with no locks: proves an HTTP thread can answer. Use it to restart a truly dead process.
• HTTP /ready with lock timeout: proves the worker can enter the critical section. Use it to stop traffic.
• HTTP /write as the probe: too heavy, and it mutates state. Do not do that in production.
• Thread dump on fail: dump frames when ready fails twice, instead of bouncing the pid and hoping.

A one-process runner

A tiny runner that makes the split visible in one terminal looks like the script below. This is a labeled reconstruction, not a service I would expose beyond localhost.

#!/usr/bin/env python3
"""Reconstruction: naive /health vs lock-aware /ready.

Run: python probe_lock_demo.py
Then curl /health, /ready, and /write in another terminal.
"""
from __future__ import annotations

import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

HOST = "127.0.0.1"
PORT = 8080
HOLD_SECONDS = 30

lock = threading.Lock()
state = {"writes": 0}


def worker() -> None:
    while True:
        with lock:
            time.sleep(HOLD_SECONDS)
            state["writes"] += 1
        time.sleep(0.05)


def lock_ready(timeout: float = 0.2) -> bool:
    acquired = lock.acquire(timeout=timeout)
    if not acquired:
        return False
    lock.release()
    return True


class Handler(BaseHTTPRequestHandler):
    def _send(self, code: int, body: bytes) -> None:
        self.send_response(code)
        self.send_header("Content-Type", "text/plain")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self) -> None:
        if self.path == "/health":
            self._send(200, b"ok\n")
            return
        if self.path == "/ready":
            if lock_ready():
                self._send(200, b"ready\n")
            else:
                self._send(503, b"lock-held\n")
            return
        if self.path == "/write":
            with lock:
                state["writes"] += 1
            self._send(200, b"wrote\n")
            return
        self._send(404, b"nope\n")

    def log_message(self, fmt: str, *args) -> None:
        return


def main() -> None:
    threading.Thread(target=worker, name="holder", daemon=True).start()
    server = ThreadingHTTPServer((HOST, PORT), Handler)
    print(f"listening on http://{HOST}:{PORT}", flush=True)
    server.serve_forever()


if __name__ == "__main__":
    main()

python probe_lock_demo.py
# terminal 2
curl -sS -m 2 http://127.0.0.1:8080/health; echo
curl -sS -m 2 http://127.0.0.1:8080/ready; echo
curl -sS -m 2 http://127.0.0.1:8080/write || echo write-timed-out

You should see /health stay fast, /ready flip to 503 while the worker holds the lock, and /write hit the curl deadline. If all three stay 200, the worker is not in the critical section yet, so wait and try /ready again.

Hour 36–48: what broke, and what I would repeat

The generated code was not evil; it answered the question I asked, which was "give me a health endpoint." I had not asked it to tell the orchestrator when writes were impossible, so it did not. I still had to own the lock topology, and a model will happily emit a 200 handler because tutorials do.

What broke in my hands:

1. I collapsed liveness and readiness into one URL that could not fail independently.
2. I let a background thread sleep inside a lock that request threads also needed.
3. I treated a green curl as proof that users could complete writes under load.
4. I asked for a probe before I drew the lock on paper, so the handler had nothing to disagree with.

What I would repeat:

1. Write the failing unit test first, even if it is twelve lines and uses no HTTP stack.
2. Reproduce on a second machine so laptop sleep cannot muddy the timings.
3. Keep /health cheap and /ready honest, then alert on the split, not on a single 200.
4. Dump frames when ready fails, instead of restarting and hoping the next boot looks cleaner.

Would I still let a coding model draft the boilerplate HTTP server after this mess? Yes, but I would paste the lock test beside the prompt and refuse to ship a probe that cannot fail that test. Green text is cheap, and a probe that cannot turn red is expensive in hours.

If you want a disposable box for the same lock-versus-probe experiment, the free server option was enough for this write-up.

Limitations

This write-up is about a process-local threading.Lock in CPython, not a distributed lock service. It does not cover database sessions, or asyncio event loops where that lock is the wrong primitive. The sleep inside the critical section is a teaching stand-in for a stuck dependency, not a backoff policy I would ship.

The tests run in one process and cannot catch a deadlock that only appears under a specific interleaving on another interpreter build. I did not collect production metrics for this note, and I am not claiming a universal probe standard for every orchestrator.

If your runtime is asyncio, a threading.Lock probe can lie in the other direction and look busy while the loop is starved. If your platform only supports one probe URL, you still need an external check that is allowed to fail independently of liveness.

Who should not use this approach

Do not copy the thirty-second hold into a real worker and call it a simulation of backpressure. Do not probe a write endpoint that charges a card, sends mail, or enqueues a one-shot job. Do not replace real tracing with a curl loop and then call that loop observability.

If you do not own the process, you cannot dump frames, and this workflow will stall in the same place my first night stalled. If your "lock" lives in Postgres or Redis, test that system with its own timeout, not with threading.Lock.acquire.

I am also not suggesting you debug every outage on a shared scratch box when the data is sensitive. The reproduction here used synthetic counters only, and it included no customer records at all.