I Debugged a Green Suite for 48 Hours. fork Had Already Copied the Mock.

Have you ever watched pytest print a wall of dots on your laptop and still felt something was off about the workers? I spent forty-eight hours chasing a helper that looked correct in the parent process and then lied inside every forked child. The suite stayed green whenever I ran it interactively, and that comfort turned into a very expensive false sense of safety. What finally broke the spell was a multiprocessing start method I had never bothered to pin down.

What I thought I was testing

I needed a small batch worker that classified records in parallel and then wrote a compact JSON summary for a later job. The sketch used ProcessPoolExecutor, a module-level function, and a tidy unittest.mock.patch around the parent call site. Did I print multiprocessing.get_start_method() during startup, or pin the method before the pool opened on any machine? I did not, because the parent returned the mocked payload every time and the dots stayed green.

The job looked like production adjacent work rather than a toy, which made the false green even harder to question later. I treated the executor like a faster map and forgot that children are not the same process with extra cores attached. Have you noticed how easy it is to test the parent and then declare the workers finished without ever logging inside them? That is exactly where this forty-eight hour mess started for me.

Hours 0-8: blaming pytest, then blaming the box

The first failure showed up when I reran the same file on a Linux shell instead of my laptop session. Spawn raised a pickle error around the mock, and I swore the remote interpreter was older, broken, or secretly running tests twice. I upgraded pytest, pinned pytest-timeout, and disabled xdist so the collection order could not shuffle the blame onto another plugin. None of that changed the split between a quiet parent and a child that refused to import my patched callable.

I then chased environment drift like it was a personality conflict between two laptops rather than a process model. Was PYTHONPATH different, was the venv stale, or had I installed unittest.mock into the wrong interpreter by accident? Those questions felt productive because they produced commands I already knew how to type. They were still the wrong questions, and they burned the first evening without a single print of the start method.

Commands I actually ran during that dead end:

python3 -c "import sys,multiprocessing as mp; print(sys.version); print(mp.get_start_method(allow_none=True))"
python3 -m pytest -q test_classify_pool.py
PYTHONPATH=. python3 -m pytest -q -p no:xdist test_classify_pool.py

The version string looked boringly current, and the test file still passed in one shell while exploding in the other. I kept rereading the traceback as a packaging problem because spawn mentioned pickle, and pickle mentioned import paths. That is a seductive misread when you are tired and the parent process still looks innocent.

Hours 8-24: making the assistant sketch louder, not clearer

I asked a coding assistant to explain the pickle error, and it cheerfully suggested making the worker a top-level function, which it already was. Then it suggested cloudpickle, a custom __reduce__, and catching BrokenProcessPool so the suite could stay green. Would those patches have hidden the inherited mock even deeper instead of forcing me to look at fork versus spawn? Yes, and I almost shipped the catch-and-retry version because it made the remote run look calm again.

I finally reproduced the split on MonkeyCode's free server and used free model access only to draft the harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script below is the actual artifact, and it fails the same way with no product involved in the loop. I needed a second Linux interpreter I did not live in all day, not a new story about models or quotas.

What I would not repeat is pasting an executor snippet and then asking any model to keep the dots green. The useful prompt, once I finally wrote it, was narrower: show me what a child process can see after fork if the parent already patched a module function. That question belongs in a lab script, not in a retry decorator.

Hours 24-40: the start method was the whole bug

Python's multiprocessing module can start children with fork, spawn, or forkserver, and ProcessPoolExecutor rides on that choice. fork copies the parent address space, including active mocks, imported modules, and locks that were never meant to travel. spawn starts a fresh interpreter and pickles the callable, so a MagicMock either fails loudly or never arrives. If you never pin the method, two honest machines can disagree without either test file being flaky in the usual sense.

I forced both methods in one file instead of arguing about laptop defaults that will keep changing across Python releases. That is the part I should have done on hour one, and it is the only comparison I now trust. Do you want the suite to prove workers run real code, or do you want it to prove the parent can patch a name and feel accomplished? Those are different tests, and fork will happily grade the second one for you.

Reproduction I wish I had on hour one

Save this as reproduce_start_method.py and run it twice. Do not mix this with pytest plugins until the printouts look boring and stable.

# reproduce_start_method.py
"""Force fork vs spawn and show a parent-only mock leaking or exploding.

Run:
  python3 reproduce_start_method.py fork
  python3 reproduce_start_method.py spawn
"""
from __future__ import annotations

import multiprocessing as mp
import sys
from concurrent.futures import ProcessPoolExecutor, as_completed
from unittest.mock import patch


def classify(record: dict) -> str:
    label = record.get("label")
    if not label:
        raise ValueError("missing label")
    return label.upper()


def run_pool(records):
    with ProcessPoolExecutor(max_workers=2) as pool:
        futs = [pool.submit(classify, row) for row in records]
        return [f.result() for f in as_completed(futs)]


def main(method: str) -> None:
    mp.set_start_method(method, force=True)
    print("start_method=", mp.get_start_method())
    records = [{"label": "ok"}, {"label": "ok"}]
    with patch("__main__.classify", side_effect=lambda record: "MOCKED"):
        try:
            print("results=", sorted(run_pool(records)))
        except Exception as exc:
            print("raised=", type(exc).__name__, exc)


if __name__ == "__main__":
    allowed = {"fork", "spawn", "forkserver"}
    if len(sys.argv) != 2 or sys.argv[1] not in allowed:
        print("usage: python3 reproduce_start_method.py [fork|spawn|forkserver]", file=sys.stderr)
        sys.exit(2)
    main(sys.argv[1])

On a Linux box that still allows fork, the first run often prints MOCKED twice and feels like a unit test victory. The spawn run typically raises a pickle error, or it calls real classify if you submit a reimportable function instead of the live mock object. Either way, you are no longer looking at the same program you mocked in the parent. Windows will refuse fork outright, which is rude and also honest.

Companion commands I now keep next to the script:

python3 reproduce_start_method.py fork
python3 reproduce_start_method.py spawn
python3 -c "import multiprocessing as mp; print(mp.get_all_start_methods())"

If fork is missing from get_all_start_methods(), skip that row instead of pretending the laptop is Linux. Forcing a method that the runtime does not implement is not a reproduction; it is just a different exception.

Decision table I now paste into the PR

I needed a table more than I needed another retry loop, because the next reviewer will ask why spawn is suddenly required.

• Parent patched a function, and you assert worker output: fork can inherit the mock and go green; spawn pickles or reimports and disagrees.
• Parent opened a DB handle, SSL context, or requests.Session: fork copies a handle children should not share; spawn does not copy it.
• Parent already started threads, then opened a process pool: fork is unsafe around locks; spawn or forkserver is the less cursed option.
• Workers must see huge copy-on-write arrays and no parent mocks: fork can still be a deliberate performance choice, but then do not mock in the parent.
• Library code calls set_start_method at import time: do not do this; only the application entry point should pin the method once.

The row that burned me was the first one, because it looks like good engineering in a code review. A green assertion on mocked worker output is not evidence that the child ran your function. It is evidence that the parent could rename a symbol before fork copied the heap.

What actually broke, after the noise

The production worker was never supposed to see the test double, and fork made that accident look like coverage. Spawn was not flaky; it was refusing to serialize an object that had no business crossing a process boundary. I had also logged only in the parent, so every successful mock looked like a successful child. That is on me, not on pytest, and not on the remote shell that finally disagreed.

A second break appeared once I stopped patching classify and started patching a module-level cache dictionary instead. Forked children saw the warm cache and skipped the branch I still needed to test, which is a quieter lie than MOCKED. Spawned children started empty and hit the real branch, which made me accuse the Linux box of dropping env vars. Same family of bug, different costume, still a start method I had not pinned.

What I would repeat next time

I would print the start method in the worker entry point, not only in the parent, because children are where the story actually happens. I would run the tiny harness under fork and spawn in CI as two explicit jobs, even if one job is allowed to skip on platforms that lack fork. I would keep worker callables as module-level functions with pickleable arguments, and I would mock inside the child or not at all. I would also refuse any patch that catches BrokenProcessPool just to keep a dashboard green overnight.

A logging snippet worth keeping:

import multiprocessing as mp
import os

def worker_boot():
    print(
        f"pid={os.getpid()} parent={os.getppid()} "
        f"start={mp.get_start_method()} ",
        flush=True,
    )

Submit worker_boot before the real map so the first lines of output name the method. If that line is missing, I no longer trust the rest of the run, including my own laptop. Forty-eight hours is a ridiculous amount of time to spend before adding one print, and I would like that not to become a personality trait.

Limitations, and who should not copy this

This workflow is a start-method lab, not a general parallelism tutorial, and it will not fix a pool that is simply too large for the machine. If you process huge numeric arrays and you chose fork on purpose for copy-on-write, do not blindly switch to spawn because a blog post got burned by a mock. If you ship a library, do not call set_start_method at import time, because that decision belongs to the application. If you are on Windows, fork is not a reproduction target, and forcing it only adds noise.

The harness also assumes you can run the same file under two methods without hidden plugins rewriting imports. It does not measure speed, and I am not claiming a benchmark, a quota, or a forever-stable default in future Python releases. Treat defaults as untrusted, pin what you mean, and keep the artifact small enough to read in one sitting. If your workers need shared mutable parent state, that is a design smell this script will expose rather than a feature to preserve.

Field notes I am keeping on the fridge

Green dots in the parent are not worker coverage, especially when fork can copy a mock you never meant to ship. Pin the start method in the entry point, log it inside the child, and run both fork and spawn before you argue about laptops. Ask whether the test proves the child ran real code, and throw away any helper whose only talent is keeping pickle errors off the screen. That is the whole forty-eight hours, compressed into a script I should have written first.