DEV Community

Nicolas VANDENBOGAERDE
Nicolas VANDENBOGAERDE

Posted on

Show HN: Solid-jobs-py – Celery alternative using Python 3.14 subinterpreters

solid-jobs-py, a Redis-backed task runner engineered around Python 3.14 subinterpreters (PEP 684 / PEP 734) to achieve true multi-core parallel execution within a single process.
the codebase uses a cross-language sidecar pattern: it delegates all Redis stream handling and atomic Lua script state checking to a companion Ruby process embedding the solid-redis driver, communicating locally via strict Unix sockets (0o600).I benchmarked it against Celery 5.6 (prefork pool) under identical hardware constraints:

  1. Memory Efficiency: At concurrency 8, the total process tree maps to 141 MiB vs 270 MiB for Celery. Marginal worker cost drops from 26.5 MiB to 10.7 MiB, resulting in 1.1x to 1.8x more jobs/s per MiB.
  2. Throughput Parity: Hits exact par with Celery on CPU-heavy iteration blocks while outperforming it by up to 3.06x on empty noop framework latency tests.
  3. Design Decisions: The README documents why I rejected MessagePack serialization (subinterpreters fallback to pure Python wheels, inflating latency to 26 µs vs 6 µs for standard standard-library https://pypi.org/project/solid-jobs-py/ https://github.com/nicolasva/solid-jobs-py

Top comments (0)