48-Hour Field Notes: Parent Health Stayed Green After Fork Copied a Held Lock

Have you ever trusted a green /health probe while every real request sat somewhere else and starved? I just burned forty-eight hours on that exact lie, and a chat window kept handing me prettier retries. The hang never showed up on my laptop, so every patch compiled in my head and still missed the process that actually forked. This is a lab notebook, not a victory lap, and it stays useful if you delete every product name.

I stood up a tiny Python worker, forced the start method in the environment, and only then let a model read traces. Why did I keep feeding laptop logs to a tool that could not see fork? Because /health answered 200 in milliseconds, and tired brains treat that as proof. The parent was healthy. The children were not.

Hours 0–8: I treated a hang as a slow client

The symptom looked boring at first, which is how these loops usually start, right? Concurrent /work calls from a probe script stalled, while parent health on another port answered immediately. I pasted the client timeout into an assistant and asked for a sturdier retry loop, because that is the reflex when you are tired and the dashboard is green.

It gave me exponential backoff, a circuit breaker sketch, and a comment about jitter. None of that was wrong in the abstract, and none of it could see a child that had copied a held threading.Lock. I ran the same probe on my laptop and watched it pass, so I assumed the remote side just needed more patience. Patience is not a fix when the lock state was cloned.

What I tried in that first block:

• Raised the client timeout from two seconds to twenty seconds
• Wrapped curl in retries instead of inspecting the lock
• Asked the model to make the handler more concurrent
• Restarted the parent and called it done when /health stayed green

Hours 8–16: Extra workers made the laptop look even healthier

Throwing processes at a mystery is a special kind of self-own, and I did it anyway. The assistant suggested a pool, which my laptop accepted because that interpreter was not reproducing the fork path I cared about. Fresh interpreters do not inherit a held lock, so /work kept answering, and I wasted a quiet afternoon celebrating a false negative. Would you have printed multiprocessing.get_start_method() before adding more children?

I did not. I checked top, saw two Python processes, and called it a pool. The parent health thread was bound to PORT+1, which is the port my probe still used, because I had wired the check to the process I started by hand. More children on the wrong start method just gave me more green lies.

Hours 16–24: The hang lived on a Linux box I finally used

I needed a second machine whose process model matched the host I actually ship to, not the one on my desk. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free server option for that box, and I used its free model access only after the server printed logs my laptop had never produced.

The same script hung on /work within a minute, while parent health on 8081 stayed cheerful. That is the whole plot, honestly. Defaults move between platforms and interpreter versions, so I stopped guessing and printed the method on both machines. I exported MP_START so the lab did not depend on whichever default that interpreter shipped with.

Commands that finally told on me:

python3 -c "import multiprocessing as mp; print(mp.get_start_method(allow_none=True))"
export MP_START=fork
export HOLD_LOCK_BEFORE_FORK=1
export PORT=8080
python3 repro_fork_lock.py

# Parent liveness, the lying one:
curl -sS -m 1 http://127.0.0.1:8081/health || echo "parent health failed"

# Child work, the one that actually matters:
curl -sS -m 2 http://127.0.0.1:8080/work || echo "work timed out"

# See who still owns the listen socket:
ss -lptn | grep 8080 || true
ps -o pid,ppid,wchan:32,cmd -C python3

If wchan sits on futex_wait while /health still smiles, you are not looking at a slow client. You are looking at a lock the child inherited in the held state. Sound familiar yet?

The lab artifact you can run

This is a constructed lab, not a production postmortem with fake graphs or invented latency numbers. Save the file as repro_fork_lock.py, then run it under fork and spawn yourself. Label every later patch as a candidate until the probe matrix moves.

#!/usr/bin/env python3
"""Lab only: parent health stays green while children deadlock after fork."""
from __future__ import annotations

import os
import sys
import time
import threading
import multiprocessing as mp
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

PORT = int(os.environ.get("PORT", "8080"))
WORK_SEC = float(os.environ.get("WORK_SEC", "0.2"))
LOCK = threading.Lock()  # created in the parent on purpose


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            body = f"pid={os.getpid()} health=parent-or-child\n".encode()
            self._send(200, body)
            return
        if self.path == "/work":
            with LOCK:
                time.sleep(WORK_SEC)
            self._send(200, b"done\n")
            return
        self._send(404, b"nope\n")

    def _send(self, code: int, body: bytes) -> None:
        self.send_response(code)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt: str, *args) -> None:
        sys.stderr.write(f"{os.getpid()} {fmt % args}\n")


def hold_lock_then_sleep() -> None:
    LOCK.acquire()
    time.sleep(30)


def run_server() -> None:
    httpd = ThreadingHTTPServer(("0.0.0.0", PORT), Handler)
    httpd.serve_forever()


def main() -> None:
    method = os.environ.get("MP_START", mp.get_start_method())
    mp.set_start_method(method, force=True)
    print(f"start_method={mp.get_start_method()} parent={os.getpid()}", flush=True)

    if os.environ.get("HOLD_LOCK_BEFORE_FORK") == "1":
        threading.Thread(target=hold_lock_then_sleep, daemon=True).start()
        time.sleep(0.2)

    procs = [mp.Process(target=run_server, name=f"w{i}") for i in range(2)]
    for p in procs:
        p.start()
        print(f"child {p.name} pid={p.pid}", flush=True)

    def parent_health() -> None:
        httpd = ThreadingHTTPServer(("0.0.0.0", PORT + 1), Handler)
        httpd.serve_forever()

    threading.Thread(target=parent_health, daemon=True).start()
    print(f"parent health on {PORT + 1}", flush=True)
    for p in procs:
        p.join()


if __name__ == "__main__":
    main()

Probe that keeps the two ports honest

#!/usr/bin/env bash
set -euo pipefail
health=$(curl -sS -m 1 -o /tmp/health.out -w "%{http_code}" http://127.0.0.1:8081/health || true)
work=$(curl -sS -m 2 -o /tmp/work.out -w "%{http_code}" http://127.0.0.1:8080/work || true)
printf "health_port=8081 status=%s body=%s\n" "$health" "$(tr -d '\n' </tmp/health.out || true)"
printf "work_port=8080 status=%s\n" "$work"
test "$health" = "200" && test "$work" = "200"

Expected matrix for this lab

MP_START	HOLD_LOCK_BEFORE_FORK	/health on 8081	/work on 8080
fork	1	200	hangs / timeout
spawn	1	200	200
fork	0	200	200

If your row does not match, you are not running the lab I ran. Stop, print the start method, and do not ask a model yet.

Hours 24–40: The model still wanted asyncio

Once the Linux traces existed, I pasted start_method=fork plus the hung curl and asked what could block /work while /health stayed warm. The first answer still reached for asyncio, then for SO_REUSEADDR, then for a bigger timeout. What is a healthcheck even proving if it never talks to the children?

I had to constrain the prompt to the evidence: a lock created in the parent, a thread that acquired it, then fork(). After that, the suggestion got closer. Do not hold non-fork-safe locks across fork, and do not let the parent advertise liveness for children it never asked. I applied the candidate on the server, not in the chat transcript, and reran the probe until the matrix moved.

What actually changed the lab:

1. Export MP_START=spawn when the app does not need fork semantics at all.
2. Or create the lock inside the child, after the start method has already run.
3. Stop the parent from answering health on a socket the children do not share.
4. Have each child touch a heartbeat path, and make readiness read those files.

Heartbeat sketch, labeled as a proposal until you wire it to your supervisor:

# Proposal: child liveness the parent cannot fake with is_alive().
import time
from pathlib import Path

beat = Path(f"/tmp/worker-{os.getpid()}.beat")

def pulse() -> None:
    beat.write_text(str(time.time()))

def children_are_live(pids: list[int], max_age: float = 5.0) -> bool:
    now = time.time()
    for pid in pids:
        path = Path(f"/tmp/worker-{pid}.beat")
        if not path.exists():
            return False
        if now - float(path.read_text()) > max_age:
            return False
    return True

I would not drop that snippet into a public ingress and call it Kubernetes-ready. It is a lab check so /health stops being a personality test for the parent.

Hours 40–48: What I would repeat

I would repeat the boring order of operations, because a chat window is a terrible debugger when it cannot see the fork. Reproduce on the machine whose start method matches production, capture the two curls, then ask a model. If the answer never mentions fork, spawn, or inherited lock state, I throw it away and tighten the prompt.

A workflow that survived the weekend:

1. Print get_start_method() and the two listen ports before any theory.
2. Run the probe against the worker port and the parent port separately.
3. Only then paste those two result blocks into a model.
4. Apply the candidate patch on the same server that hung.
5. Keep the matrix in the repo so the next forty-eight hours start shorter.

If you already have a Linux box, you do not need a new product to run this lab. I used MonkeyCode because the free server option and free model access sat in the same loop I was already running; treat that as convenience, not as a requirement.

Limitations, and who should skip this

This approach will waste your time if you cannot run extra processes, or if the hang lives in a platform you cannot curl. A free server is not your production cgroup, and a free model will not read /proc unless you paste the output. I am not claiming model names, quotas, hardware, duration, or benchmarks, because I cannot honestly attest to those here.

Do not send secrets, customer traffic, or private keys into any model, free or otherwise. Do not copy this parent-health pattern into a real service and call it readiness. If your workers already pin spawn everywhere, this particular footgun may not be yours, and you should hunt inherited file descriptors instead of locks.

Fork plus threads remains sharp even after this lab. The artifact teaches you to notice a lying parent, not to invent a multiprocess framework on a deadline. If your incident is a network partition, this notebook will not save you, and you should stop asking a model to rewrite locks you have not proven are held.