DEV Community

Serguey Asael Shinder
Serguey Asael Shinder

Posted on

Your Worker Pool Relied on fork Copying Everything and Python 3.14 Starts Workers Fresh

The nightly job
loaded every price
into one big dictionary,
two gigabytes of it.

PRICES = {}

At the top of the module.

Then main filled it:

PRICES.update(load_prices())

Then it split the order lines
across eight worker processes
and each worker
looked up prices in that dictionary.

It never passed the data in.

It never had to.

On Linux
it has worked for years.

That is because of how
the workers were made.

The default start method
on Linux was fork.

fork copies the parent process
as it is at that moment,
memory and all.

The dictionary was there,
so the workers had it,
almost for free.

Python 3.14
changed that default.

On Linux and other Unix systems
apart from macOS,
new process pools
now start with forkserver.

A small server process
starts early,
before your main has run,
and every worker
is forked from that clean server,
not from you.

Your dictionary is not in it.

The workers start
with the module as it is
on import,
and PRICES is an empty dictionary.

Every lookup misses.

Everything you pass to a worker
now has to be pickled.

Lambdas and local functions
stop working as targets.

And code without
if name == "main":
starts doing strange things.

None of this is new.

macOS has used spawn
since Python 3.8.

Windows never had fork.

Your job only ever worked
on one platform,
by accident.

The reason for the change is real.

Forking a process
that is running threads
copies locks
other threads were holding,
and the child can hang forever.

Python 3.12 already started warning
when that happens.

So do the work now,
before the upgrade.

Pass the data in explicitly,
through an initializer,
or load it inside each worker.

Put the main guard
where it belongs.

Then run your tests
with the start method
set to forkserver
on the version you already have.

It has existed
since Python 3.4.

If you truly need fork,
ask for it by name,
with a comment explaining why.

A default
is a decision someone else made.

Find the ones you have been
quietly leaning on.

– Serguey Asael Shinder

Top comments (0)