The Pool Joined Forever. I Had Never Printed the Start Method.

Have you ever watched a parallel job hang only after you moved it off your laptop? I spent two nights staring at a worker pool that printed one line, then went silent. Locally the same tree finished in a few seconds and I kept asking the wrong question. Was the free server slow, or was I shipping macOS behavior onto Linux without noticing?

This is a 48-hour field log, not a product tour. I used a free coding model to draft a tiny batch job, then ran the same checkout on a free server. The hang was real. The model was a side character. The bug lived in multiprocessing start methods and a lock I created in the parent.

Hour 0: a boring batch that looked fine

I needed to hash a folder of JSON receipts and write a manifest. Nothing glamorous, no GPU, no queue. I asked a free model for a four-worker pool because the laptop version felt sequential and I wanted wall-clock time down. The first draft looked like every tutorial I have ever skimmed.

# proposed snippet — I did not run this blindly
from multiprocessing import Pool, Lock
from pathlib import Path
import hashlib, json, os, sys, time

print_lock = Lock()

def hash_one(path: str) -> dict:
    data = Path(path).read_bytes()
    digest = hashlib.sha256(data).hexdigest()
    with print_lock:
        print(f"hashed {path} {digest[:12]}", flush=True)
    return {"path": path, "sha256": digest}

def main() -> None:
    files = [str(p) for p in Path("receipts").glob("*.json")]
    print(f"count={len(files)} pid={os.getpid()}", flush=True)
    with Pool(4) as pool:
        rows = pool.map(hash_one, files)
    Path("manifest.json").write_text(json.dumps(rows, indent=2))
    print("wrote manifest.json", flush=True)

if __name__ == "__main__":
    main()

On my Mac it printed every file and exited zero. Why would I distrust that? I copied the tree to the free server, installed the same unpinned dependencies, and started the script. It printed count=40 pid=... and then nothing. No traceback. No worker line. Just a process that refused to die.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access to draft the worker script, then reproduced the hang on MonkeyCode's free server option. I am describing that workflow as a user of those two availability claims, not as a benchmark of unnamed models or hardware.

Hours 1–8: I blamed the network, then the disk

Was the volume slow? I ran time dd if=/dev/zero of=scratch.bin bs=1M count=32 and it returned quickly enough that I felt silly. Was DNS involved? There is no DNS in a SHA-256 loop. I still checked it, because hanging jobs make you superstitious.

What I tried, in order:

1. Added flush=True everywhere, because stdout buffering is a classic liar.
2. Reduced the pool to one worker, which still hung after the parent print.
3. Replaced pool.map with a plain for loop in the parent, which finished immediately.
4. Ran ps -o pid,ppid,stat,wchan,cmd and stared at futex_wait without understanding it.
5. Asked the free model why a pool would freeze, and it suggested timeouts, RAM, and "try more workers."

That last suggestion was the worst hour. More workers on a deadlocked parent is how you turn a quiet hang into a louder hang. I still have the chat scroll. I would not repeat that prompt.

Hours 9–20: the logs were telling the truth

I finally printed the one line I should have printed before any pool existed. Defaults differ across platforms and across Python versions, so I stopped assuming Linux still meant fork in my head.

import multiprocessing as mp
import platform, sys

if __name__ == "__main__":
    print("python", sys.version.split()[0], flush=True)
    print("platform", platform.platform(), flush=True)
    print("start_method", mp.get_start_method(), flush=True)
    print("available", mp.get_all_start_methods(), flush=True)

Laptop output, condensed: start method spawn. Free server output, condensed: start method fork. Same repo. Same script. Different inheritance rules. Have you checked that on your CI image this week, or are you still trusting the tutorial that never printed it?

Here is the failure in one sentence that I wish I had written on a sticky note. On fork, child processes inherit the parent's locks in a locked or inconsistent state. On spawn, they do not inherit that live lock; they import a fresh module. I created print_lock in the parent at import time, then the forked workers blocked forever trying to acquire a lock whose owner died in another address space.

The sequential loop never touched the pool, so it never inherited the mess. That is why hour 3 "proved" the hashing was fine and taught me nothing useful.

Hours 21–32: a tiny repro I can rerun

I threw away the receipts folder and wrote a repro that deadlocks without any JSON. If this hangs on your machine, you are on a start method that inherited the parent lock. If it prints child-ok and exits, you are not.

# repro_lock.py — labeled experiment, keep the timeout
import multiprocessing as mp
import os, sys, time

lock = mp.Lock()

def worker(n: int) -> str:
    # If this line never prints, the inherited lock is the suspect.
    with lock:
        return f"child-ok n={n} pid={os.getpid()}"

def run(method: str, timeout: float = 5.0) -> str:
    ctx = mp.get_context(method)
    # Fresh lock in this context so we do not mix methods.
    local_lock = ctx.Lock()

    def inner(n: int) -> str:
        with local_lock:
            return f"{method} n={n} pid={os.getpid()}"

    with ctx.Pool(2) as pool:
        async_result = pool.apply_async(inner, (1,))
        return async_result.get(timeout=timeout)

if __name__ == "__main__":
    method = sys.argv[1] if len(sys.argv) > 1 else mp.get_start_method()
    print("trying", method, flush=True)
    try:
        print(run(method), flush=True)
    except Exception as exc:
        print(type(exc).__name__, exc, flush=True)
        sys.exit(2)

Commands I actually kept in the notes:

python -c "import multiprocessing as mp; print(mp.get_start_method()); print(mp.get_all_start_methods())"
python repro_lock.py spawn
python repro_lock.py fork
python repro_lock.py forkserver

On the free server, spawn returned child-ok inside the timeout. fork hit TimeoutError once I stopped waiting all night. forkserver behaved like a grown-up and did not inherit the import-time lock the same way. I am not publishing timings, because I did not run a controlled benchmark and I will not invent one.

The original hang was even cheaper to trigger than this file. Import-time Lock() plus Pool plus fork is enough. Why did the model write it that way? Because every snippet on the internet creates a global lock, then a pool, then a print.

Hours 33–48: what I pinned, and the test I will keep

I stopped asking the model for "a faster pool." I asked it, after I understood the bug, to add an explicit start method and to refuse import-time locks. Then I wrote a test that does not care which cloud box I am on.

# test_start_method.py
import multiprocessing as mp
import os, sys, pytest

@pytest.mark.parametrize("method", [m for m in ("spawn", "forkserver") if m in mp.get_all_start_methods()])
def test_pool_does_not_hang(method):
    ctx = mp.get_context(method)
    lock = ctx.Lock()

    def work(x):
        with lock:
            return x * 2

    with ctx.Pool(2) as pool:
        assert pool.apply_async(work, (21,)).get(timeout=8) == 42

def test_parent_records_method(tmp_path):
    # Guard against silent default drift after a Python upgrade.
    recorded = mp.get_start_method()
    (tmp_path / "method.txt").write_text(recorded)
    assert recorded in mp.get_all_start_methods()

The production change was three lines in main, not a new framework.

def main() -> None:
    # Pin after the feature freeze, never at import time in a library.
    mp.set_start_method("spawn", force=True)
    files = [str(p) for p in Path("receipts").glob("*.json")]
    with mp.Pool(4) as pool:
        rows = pool.map(hash_one, files)

I still dislike force=True in libraries. In a top-level script that I own, it is honest. The free server then printed every hash line and wrote manifest.json. The laptop still passed. That is the only result I am willing to claim.

Decision table I taped above the desk

Symptom on the remote box	What I used to guess	What I check now
Parent prints, workers never print	"disk is slow"	get_start_method() and wchan
Works in a for loop, dies in Pool	"pool size is wrong"	locks created before Pool
Works on macOS, hangs on Linux	"the server is weaker"	spawn versus fork inheritance
Model says add timeouts and workers	"maybe it is load"	one pinned start method plus a 5s get
Fine after a rewrite that removes prints	"logging was expensive"	whether prints took a lock

What broke, what I would repeat

What broke was not "AI wrote bad Python" in the abstract. What broke was an unpinned start method plus a parent lock plus my habit of trusting a green local run. The free model repeated a tutorial shape. I repeated it onto a Linux box. The hang was deterministic once I stopped treating it like weather.

What I would repeat:

• Print sys.version, platform.platform(), and mp.get_start_method() in the first log line of any parallel job.
• Give AsyncResult.get a timeout in experiments so a deadlock becomes an exit code.
• Keep a five-file receipts fixture so I am not debugging production data while I debug process startup.
• Ask the model for a repro that hangs on purpose, not for a pep talk about RAM.
• Run the same checkout on the free server before I believe a laptop green bar.

What I would not repeat: adding workers, adding retries, or letting the model rewrite the hasher while the parent was still holding a lock. Those patches optimized a path that was never running.

Limitations, and who should skip this

This approach is for small batch scripts you can kill without hurting anyone. It is not a production scheduler. A free server is a shared, constrained place to reproduce platform drift, not a place to hide a fleet of CPU workers. I am not claiming uptime, cores, quotas, or model quality, because I was not measuring those.

Do not use this pattern if any of the following is true:

• You need in-memory fork copy-on-write for a huge parent heap and you already audit every lock.
• You ship a library and call set_start_method for your users.
• Your workers must inherit sockets, tracemalloc hooks, or logging handlers that spawn will not copy.
• You cannot install a second Python start method on the image.
• You are debugging a hang that already has a traceback; start methods are the wrong first hypothesis then.

spawn also costs import time, because each child re-imports your module. If your module talks to the network at import time, you just bought four extra client connections. Pinning the start method does not make side effects at import safe. It only makes the lock story predictable.

Python's default start method has already moved once on macOS, and maintainers keep warning that fork is a sharp edge with threads. I am not going to freeze a default in this article as if it were a law. The whole point of the field notes is to print the method on the box you actually run.

If you keep a similar two-night log, tell me which start method you pinned and whether forkserver surprised you. I still reread the futex_wait line and wince, which is probably the remaining educational value.