48-Hour Field Notes: The Pool Hung After Fork. My Laptop Used Spawn.

Have you ever watched a process pool freeze while the parent kept logging like nothing at all was wrong? I spent two days blaming the remote runtime before I finally printed one extremely boring diagnostic line. The two machines did not disagree about Python versions in any way that looked interesting at first. They quietly disagreed about how brand new worker processes were actually being born on each machine.

Do you actually know your start method right now, or are you still assuming the runtime you use at your desk? I assumed, and that assumption survived code review, extra timeouts, and a ridiculous amount of extra logging. The extra logging is what made the hang easier to trigger, which felt deeply unfair at the time. Why would more logs make a worker quieter?

What I thought I was debugging

The symptom looked like a stuck job, not like a multiprocessing lecture from the standard library documentation. A parent process accepted work, submitted a batch into ProcessPoolExecutor, and then sat there forever with a still-warm event loop of logs. The child never printed the first line inside the worker function, so I kept staring at timeouts, CPU, and “the server is slow.” Was the worker even alive, or had it died before my first logger.info?

I wrote the same kind of field note I always write when two environments disagree. Laptop green, second machine hung, no useful traceback, exit code missing in action. I did not have a customer name, a dashboard screenshot, or a heroic outage metric to wave around. I had a tiny script, two interpreters, and a bad habit of trusting the machine that felt faster under my fingers.

What I tried for two days

Here is the messy list, because field notes that skip the dumb attempts are just marketing copy.

1. I increased the future timeout and then blamed the remote host for being “just slower than my laptop.”
2. I sprinkled logging.info in the parent, the worker, and a finally block that never ran on the child.
3. I printed os.cpu_count() even though this hang was not an oversubscription story at all.
4. I pinned max_workers=1 so the pool could not possibly fight itself, and it still froze.
5. I restarted the remote process a few times, which cleared a hung child and taught me nothing durable.
6. I searched for a deadlock in my locks, while the standard library already owned a lock I had copied.

Did any of that help? It helped me write a longer wrong story. The parent kept talking because the parent still held a healthy logging lock. The child went quiet because it was born in the middle of that lock story. Have you noticed how often “add more logs” is the step that reproduces a fork bug on demand?

The one line that split the machines

The useful command was not a profiler. It was this, run in both places without any of my application code in the way.

python -c "import multiprocessing as mp; print(mp.get_start_method())"
python -c "import multiprocessing as mp; print(mp.get_all_start_methods())"
python -c "import sys,platform; print(sys.platform, platform.python_implementation(), sys.version.split()[0])"

One interpreter answered spawn. The other answered fork. That is the whole plot, and I had been decorating the wrong mystery for hours. Spawn pickles the callable, starts a fresh interpreter, and reimports your module under a new __main__ guard. Fork copies the parent address space, including locks that a logging handler might currently hold in another thread.

I eventually reran the same tiny file on MonkeyCode's free server option, because I needed a second interpreter that was not my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I pasted the two start-method dumps into MonkeyCode's free model access, asked it to list fork-safety problems, and then I verified every claim against the multiprocessing start-method docs plus a local repro. The product was a second machine and a skeptical rubber duck, not evidence that the pool was healthy.

A tiny lab that hangs on purpose

This is a reconstructed lab, not a production postmortem with invented traffic numbers. If you run it on a Linux fork context while the parent is logging from a background thread, the child can stall before it writes anything. Label this as an unexecuted-on-your-box example until you actually run it.

# lab_fork_logging.py
# Lab reproduction: fork can copy a held logging lock into the child.
import logging
import multiprocessing
import threading
import time

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(processName)s %(message)s",
)
log = logging.getLogger("lab")


def child():
    # If this line never appears, the child likely inherited a held lock.
    log.info("child reached the first log line")


def spam_parent():
    while True:
        log.info("parent still logging")
        time.sleep(0.001)


def main():
    method = multiprocessing.get_start_method()
    log.info("start method is %s", method)
    t = threading.Thread(target=spam_parent, daemon=True)
    t.start()
    time.sleep(0.05)
    proc = multiprocessing.Process(target=child, name="lab-child")
    proc.start()
    proc.join(timeout=5)
    log.info("alive=%s exitcode=%s", proc.is_alive(), proc.exitcode)
    if proc.is_alive():
        proc.terminate()
        log.info("terminated hung child after join timeout")


if __name__ == "__main__":
    # Force the interesting path on Linux. Windows cannot honor fork.
    multiprocessing.set_start_method("fork", force=True)
    main()

What should you look at after you run it? If join times out and exitcode is None, stop talking about HTTP timeouts. If spawn makes the child log immediately, stop talking about “the remote host is flaky.” Can a one-file lab replace a distributed trace? No, but it can stop you from rotating the wrong knob for another night.

A spawn-shaped cousin of the same bug is not a hang. It is a PicklingError, a reimport that skips parent-only mutations, or a worker that cannot see a nested function. I keep this dump next to the hang lab because I keep mixing the two failures in my head.

# dump_context.py
import multiprocessing
import os
import sys

def work(n):
    return n * n

if __name__ == "__main__":
    ctx = multiprocessing.get_context()  # default context for this platform
    print("platform", sys.platform)
    print("start_method", ctx.get_start_method())
    print("all_methods", multiprocessing.get_all_start_methods())
    print("pid", os.getpid())
    with ctx.Pool(2) as pool:
        print("pool_ok", pool.map(work, range(4)))

A decision table I wish I had on hour one

I now keep this table in the same directory as the lab file, because my memory is optimistic and the runtime is not.

• Hung child, parent still logging, no traceback: print get_start_method(); if it is fork, treat logging locks and other non-fork-safe locks as guilty until proven otherwise.
• PicklingError or AttributeError on a nested callable: you are on spawn or forkserver; move the worker to module top level and keep the if __name__ == "__main__" guard.
• Worker missing a mutation you made in the parent after import: spawn reimports; fork copied old memory; stop using hidden global state as a message bus.
• Works with max_workers=1 in a debugger but fails under load: you probably added threads, logging, or a client session before the first fork.
• Windows-only developer, Linux-only server: the start methods will not match unless you set one explicitly and then test that choice on both sides.

Would I trust this table without running the dump? Not anymore. The dump is cheaper than another night of staring at parent logs that are lying by omission.

The fix I would actually repeat

I do not want a clever fork trick. I want the same process birth on every machine I claim to test.

import logging
import logging.handlers
import multiprocessing
import queue
from concurrent.futures import ProcessPoolExecutor

def configure_parent_logging():
    log_queue = multiprocessing.Queue(-1)
    listener = logging.handlers.QueueListener(log_queue, logging.StreamHandler())
    listener.start()
    root = logging.getLogger()
    root.handlers.clear()
    root.addHandler(logging.handlers.QueueHandler(log_queue))
    root.setLevel(logging.INFO)
    return listener

def work(n):
    logging.getLogger("worker").info("work %s", n)
    return n * n

def main():
    multiprocessing.set_start_method("spawn", force=True)
    listener = configure_parent_logging()
    try:
        with ProcessPoolExecutor(max_workers=2) as pool:
            print(list(pool.map(work, range(8))))
    finally:
        listener.stop()

if __name__ == "__main__":
    main()

Why spawn in the sample instead of “just close your handlers”? Because I can explain spawn to the next reader without pretending every lock in every dependency is fork-safe. QueueHandler is still worth using, because child processes should not fight the parent over a stream. forkserver is a reasonable third option on Linux, and I would still print it instead of assuming it.

Limitations, and who should skip this

This notebook is not a benchmark, and it is not a claim about anyone’s hardware quota, model name, or uptime. set_start_method can only be forced once, and test runners that already created processes will argue with you. Windows cannot fork, so a Linux-only hang lab will not teach a Windows-only laptop anything until you add a second environment. Copy-on-write tricks, shared ctypes memory, and “we fork after warming a huge cache” are real techniques, and this article is the wrong cheerleader for them.

Skip this approach if your workers must inherit live sockets, live database handles, or a giant in-memory parent cache that you refuse to rebuild. Skip it if you need to debug native extensions that register at-fork hooks you do not own. Skip it if you are trying to make pickle accept a lambda; that is a different mess, and spawn will just fail louder. Also skip any model suggestion you cannot recheck against CPython docs and a file you actually ran.

What would I repeat tomorrow morning without drama? Print the start method first, pin it in the entrypoint, and keep a lab that fails in five seconds. Would I add more parent logs before that print? Not unless I want another quiet child. If you run the dump on a second host, I want to hear which field flipped first.