I parallelize Python loops with two comments - and guarantee identical results published: true

Soumik — Fri, 24 Jul 2026 06:24:18 +0000

Python has your cores. It just doesn't use them. A CPU-bound for loop runs on one core while the rest sit idle, and the usual fixes (multiprocessing, concurrent.futures, joblib) ask you to restructure code, manage pools and pickling, and reason about what's actually safe to run in parallel. That's real work, and it's easy to get subtly wrong.

I wanted something else: mark a loop, and if it's safe and worth it, run it in parallel; otherwise run it exactly as before. No rewrite. No wrong answers. That's Lucen.

The whole API is two comments

# LUCEN START
for i in range(len(rows)):
    out[i] = expensive(rows[i])
# LUCEN END

Run the file and that loop runs across your cores:

pip install lucen
lucen run yourscript.py

Delete the two comments and the file is ordinary Python again, byte for byte.

The one guarantee

Most "fast Python" tools ask you to trust them. Lucen makes exactly one promise, with no tiers and no opt-out:

A parallel run is bit-identical to the same file run as plain sequential Python, floating-point results and container insertion order included.

Not "close." Not "for the mode you enabled." Identical. Floats reduce in sequential order so the last bit matches; lists and dicts keep the exact order they'd have single-threaded. If Lucen can't guarantee that for a loop, it doesn't parallelize it; it runs sequentially and tells you why.

How it decides

Two questions, both answered before anything runs in parallel:

Is it safe? Lucen analyzes the loop body (reads, writes, aliasing, and the shape of any cross-iteration dependency) and only parallelizes when it can prove iterations don't interfere in a way it can't reconcile. When it can't read a called function's source, you vouch for it with # LUCEN TRUST; otherwise it stays conservative.
Is it worth it? Parallelism costs dispatch, and on some interpreters, pickling. Lucen probes the real per-iteration cost and only parallelizes when the speedup beats that overhead. A trivial loop stays sequential, because making it "parallel" would make it slower.

It picks the backend for you

The right way to run parallel Python depends on the interpreter, not your code:

On a normal (GIL) build, CPU-bound Python can't speed up with threads, so Lucen routes to processes.
On free-threaded (no-GIL) CPython 3.13/3.14, threads finally parallelize Python, so Lucen routes to threads, skipping pickling and subprocess cost entirely.

Same two comments either way. On my machine, CPU-bound loops land around 3 to 4.6x: via processes on 3.11 and via threads on 3.14t, with no code change.

Why I trust it enough to call it 1.0

Parallelism bugs are the worst bugs: nondeterministic, rare, and disqualifying for a tool whose entire pitch is "identical results." So correctness is the project:

Differential testing: generated programs run parallel vs. sequential, compared bit-for-bit.
Property testing with Hypothesis, plus whole-program and front-end fuzzing.
TLA+ specifications for the privatize-and-commit and wavefront protocols, model-checked.
An optional Rust core accelerates orchestration, with a pure-Python fallback guaranteed to produce identical results, so pip install always works and the answer never depends on whether the native core loaded.

Apache-2.0, published with trusted publishing and signed releases.

Try it

pip install lucen
lucen run yourscript.py

Mark a slow loop, run it, then delete the comments and confirm the output is identical. That is the whole experience, and the point.

Repo, benchmarks, and design docs: https://github.com/fcmv/lucen

I'd love feedback, especially from anyone running CPU-bound Python in anger. What would you point it at?