Btw , here’s a fun fact: while some people build projects on 64-core monsters with 128 GB of RAM, this entire algorithm—every test and every overflow fix in codec.rs—was written, compiled, and debugged right on a smartphone screen within the Termux terminal. To be honest, it was a pretty hardcore experience, but if the code compiles and runs blazingly fast on mobile hardware, it means the architecture is truly lightweight and clean.
Every data pipeline I’ve ever profiled eventually slams into the same two walls: memory allocation storms inside shuffle buffers, and the godawful latency of integer division in the entropy coder. It doesn’t matter if you’re feeding a GPU training loop, shipping terabytes of observability logs, or ingesting real-time sensor telemetry—you are constantly forced to pick a poison. You either accept the weight of dynamic memory, or you hardcode static probability tables and leave compression ratio on the floor. I got tired of the compromise. So I built something that refuses to play that game.
The Bottleneck: Memory, Division, and Dirty Compromises
Modern data infrastructure is a ballet of tight loops. In theory, we process bytes and immediately forget them. In practice, every general-purpose compressor inserts invisible walls. A standard deflate or Zstd stream allocates internal state, hash tables, and sliding windows. For a single file, that’s fine. For a million concurrent gRPC streams or an Apache Arrow flight, those allocations turn the heap into molasses.
Then comes the entropy coding stage. If you peek inside a classic arithmetic coder or even a fast Huffman variant, you’ll find a hard integer division on the critical path. Dividing by a dynamic total probability—range / total—is not just a few cycles; it flushes pipelines, introduces unpredictable latency, and on weaker cores (think ARM Cortex-M or an energy-sipping RISC-V) it can dominate execution time. The industry has normalized this. We lean on static dictionary compressors for speed, or we pay the ratio penalty. We pretend that a 7% improvement in compression isn’t worth a 40% CPU hit. I reject that dichotomy.
Axioma: Not a Tool, but a Mathematical Engine
axioma is my attempt to burn the compromise to the ground. It is not a command-line utility with a clever flag set. It is a universal data compression algorithm—a pure, zero-allocation, self-adapting mathematical model. The entire machinery is designed around a single audacious claim: you can have an arithmetic coder with no static tables, no dynamic memory allocation, and absolutely no integer division on the hot path, while still letting the model learn the source distribution online and converge to optimality.
I named it axioma because the architecture rests on a handful of self-evident truths about information theory and computer arithmetic. If we fix the total probability mass to a power of two, division becomes a right shift. If we keep a dynamic manifold of symbol probabilities that enforces that invariant under every update, we never need to renormalize with a division. If we eliminate all heap interaction, the compressor becomes a pure function of a byte stream and a tiny stack-allocated context.
How It Works: No Static Tables, Zero Division
The heart of axioma is a classical arithmetic coder with a twist that changes everything.
The Probability Manifold
Instead of a static frequency table or a precomputed Huffman tree, axioma maintains a dynamic probability manifold of 256 symbols. Each symbol’s weight is an integer, and the sum of all weights is rigorously kept at 4096 (2¹²). This is the invariant: ∑ weight[s] = 4096 at all times.
Why 4096? Because it turns the arithmetic interval scaling into a single right shift. Traditional arithmetic coding computes:
new_range = (range * prob) / total
With total = 4096, that becomes:
new_range = (range * prob) >> 12
No division. No runtime variable divisor. The interval update collapses to a multiply and a shift, both of which modern CPUs swallow in 1–2 cycles without pipeline stalls. The low boundary update is equally trivial: low += (range * cum_prob) >> 12.
Division-Free Adaptation
The real magic is how the model learns without ever breaking the invariant or introducing a division. When a symbol is encoded, its weight must increase, but the total sum must stay 4096. A naive approach would require subtracting probability mass from every other symbol proportionally—a division-heavy nightmare.
axioma uses a lazy, shift-based rebalancing. The encoder keeps a small set of integer accumulators and a periodic rescaling policy. Once a symbol’s weight exceeds a carefully chosen threshold, the entire manifold is rescaled via an integer right shift (a decay operation). Because we rescale by powers of two, the total sum temporarily contracts, and a constant-time correction adds back the missing mass to satisfy the 4096 invariant—all without a single division. The amortized cost is negligible, and the critical encoding path sees only a conditional increment and a few bitwise operations.
The result: the model converges online to the source’s empirical distribution. Feed it JSON, and the curly braces and quotation marks will naturally sink to fewer bits. Feed it protobuf, and its varint patterns will be rewarded. No pre-training, no hardcoded tables.
Zero Allocations, Literally
The entire encoder/decoder state fits in a small C structure with fixed-size arrays. No malloc, no calloc, no sliding window heap extensions. You can instantiate a compressor context on the stack, compress a stream, and drop it. This makes axioma viable for hard real-time systems, kernel modules, and WASM sandboxes where a heap allocation is either forbidden or a performance sin.
Universal by Design: Deterministic, Parallel, and Portable
Because axioma carries no static dictionary and its initial probability manifold is perfectly uniform (every symbol starts with a weight of 16, summing to 4096), compression is stream-deterministic. The same input bytes always produce the same output bytecode, no matter the platform. That property is gold for reproducible data pipelines and content-addressable storage.
The architecture is also embarrassingly parallel at the stream level. Since each block of data can start with a fresh manifold state, you can slice a multi-gigabyte file into chunks, compress each chunk independently on separate cores, and concatenate the results. A decoder simply resets its context at chunk boundaries. On a 128-core server, this turns compression into an almost linearly scalable map operation. The same algorithm that sips power on a Raspberry Pi 4 can saturate a NUMA monster.
This is the proof I wanted to put into the world: a division-free arithmetic coder with an online self-tuning model is not just an academic toy. It is practical, fast, and universal—from a Cortex-M0 sensor node to a Milan-X server chip.
An Open Invitation (and a Dose of Honesty)
The core design and a reference implementation are fully open source. I’m putting them in your hands because I believe the next generation of data infrastructure—streaming columnar formats, edge telemetry protocols, embedded ML feature stores—deserves compression that doesn’t force a trade-off.
Here’s the reality: at the time of writing, the repository’s documentation is still mostly in Russian. It’s my native language, and translating the mathematical commentary into crisp English is taking time. I am actively tracking down edge cases and squashing bugs. If you look inside the encoder’s renormalization loop, you might find a comment that reads more like a cry for help than a specification. That’s intentional transparency. I don’t want to pretend this is a polished product. It’s a deep systems experiment that works, and I need more eyes on the architecture.
I’m inviting you to tear it apart.
· Systems engineers: review the arithmetic coding pathways, the shift-based rescaling, and the zero-alloc guarantees. Prove me wrong on a corner case.
· Mathematicians and information theorists: explore the probability manifold convergence. Can the rescaling policy be tuned for even faster adaptation? Is there a tighter invariant?
· Compiler and embedded wizards: port the inner loops to SIMD or bare-metal and show me how many cycles it really takes on a Cortex-M0.
You’ll find the code, the white-paper drafts, and the growing test suite on GitHub:
https://github.com/axmalgorithm-ops/axioma
Let’s stop accepting compression that makes us choose between speed and ratio. Let’s build infrastructure that treats the mathematical beauty of information theory as a first-class systems primitive. Dive into the code, open an issue, send a pull request—even if it’s just a fix for my broken English comments. I’ll be there, in the threads, hacking on the next iteration.
Murad Abbasov
Systems engineer, compression nerd, and perpetual debugger
Top comments (0)