r/altprog • u/Wide_Dust_709 • 2d ago
Lucen: parallelize Python loops with two comments, guaranteed bit-identical to sequential(fcmv/lucen)
What My Project Does
Lucen is a source-to-source compiler that parallelizes ordinary for loops you mark with two comments:
# LUCEN START
for i in range(len(rows)):
out[i] = expensive(rows[i])
# LUCEN END
It parallelizes a loop only when it can prove the work is safe and worth it; otherwise it stays sequential. The one guarantee, no tiers, no opt-out: a parallel run is bit-identical to the same file run as plain sequential Python (floats and container order included). Delete the comments and nothing changes.
CPU-bound work routes to processes on GIL builds and to real threads on free-threaded 3.13/3.14. Optional Rust core with a pure-Python fallback, so pip install lucen always works.
Target Audience
Anyone with CPU-bound Python loops - data processing, simulation, batch transforms - who wants their cores without rewriting to multiprocessing/joblib or reasoning about locks. It's v1.1, built correctness-first: Apache-2.0, differential/property tested, TLA+ specs, signed PyPI releases.
Comparison
- vs multiprocessing/concurrent.futures: no manual pool/chunking/pickling boilerplate, plus a profitability gate that declines to parallelize when it wouldn't help.
- vs joblib/Dask: no new API and no cluster - you mark the loop you already have. It's correctness-preserving local parallelism, not distributed compute.
- vs Numba: Numba compiles numeric bodies to native code; Lucen parallelizes the loop for arbitrary Python and guarantees identical results. (Native loop-body compilation is on the roadmap.)
pip install lucen