A few weeks ago my friend @u84u and I (@ronak-create) had a simple, slightly annoying question: if LLMs get 30-45% smaller representations of text by tokenizing it into subwords instead of raw bytes, why don't we tokenize text before handing it to a byte-level compressor like LZMA or zstd?
It felt like the kind of idea someone must have already tried and quietly dropped. So instead of writing a blog post about the idea, we built the harness to actually test it, and the result is parmar — a subword-tokenization pre-filter for byte-level compressors, plus a fairly paranoid benchmarking rig to check whether the idea holds up at scale.
Short version: it works, but for a narrower reason than we expected, and the harness told us that just as clearly as it told us the headline number.
The hypothesis
Byte-level compressors like LZMA2 and zstd find repeated patterns inside a fixed-size sliding "dictionary window," measured in bytes. Our premise: if you replace UTF-8 prose with BPE token IDs — the same tokenization used to feed LLMs — before compressing, the token stream is roughly 45% smaller than the source text. A 64 MiB dictionary window that normally covers ~64 MB of prose can now cover roughly twice as much prose once that prose has been pre-shrunk.
The catch is that this only becomes testable once your corpus is bigger than the compressor's window. On a 5 MB file, everything already fits inside the window, so there's no expansion to measure and pre-tokenization buys you basically nothing. That's why a single ratio number on a small test file is close to meaningless here — what actually matters is the curve of (parmar ratio − raw-byte ratio) as corpus size grows, and whether that curve keeps climbing.
So that became the deliverable: not "does it compress smaller," but "does the advantage widen with scale, and if so, why."
What we actually built
parmar is two things bolted together:
-
The pipeline (
parmar_core.py,parmar.py) — tokenize text withtiktoken, pack the token IDs (LEB128 or a fixed 2-byte width), and stream the result straight into anxz/zstd/gzip/bzip2subprocess. Nothing is ever fully materialized in memory — chunks are read, tokenized, packed, and piped into the compressor's stdin, and its stdout is the archive. -
The harness (
matrix.py,run_cell.py,analyze.py) — a matrix runner that sweeps tokenizer × packing × backend × transport × chunking × threading, on a real corpus, across four size tiers.
The corpus is PG-19 (Rae et al., 2019's long-range-sequence-modelling dataset), built out to four tiers from 64 MB to 4 GB, about 10,600 documents. A literal cartesian product of every axis is on the order of 8,000+ valid cells per tier — weeks of runtime — so we split it into a 51-cell "ratio grid" that isolates the axes that actually move compression ratio, and a separate one-factor-at-a-time sweep for the axes that should only affect speed. Ratio is still logged on every OFAT cell too, so if a "speed-only" axis quietly moves the ratio, that shows up as a contradiction instead of getting averaged away.
Every single decompression is actually executed and checked against a sha256 written into the archive footer. If that check fails, the cell's ratio is thrown out and reported separately, not folded into the averages. Across 452 matrix cells: 452 verified round trips, zero failures.
What we found
The gap does widen with corpus size — but only for backends with a big enough window, and it plateaus. On gzip's tiny 32 KiB window, the +15% advantage is completely flat from 64 MB to 4 GB, because the window is already saturated at every tier — that's pure "denser representation," no window effect at all. On LZMA and zstd variants with multi-megabyte windows, the gap climbs from 64 MB up through roughly 1 GB and then flattens out once the corpus is well past the dictionary size. One backend, zstd --long with its 2 GiB window, was still climbing at the 4 GB tier — because 4 GB is only 2x its window, so it hadn't left that regime yet.
Having gzip in the matrix as a tiny-window control turned out to be the thing that separated two effects we'd originally lumped together: raw representation density (tokens are just a denser way to write English than UTF-8) versus window expansion (a smaller stream lets the same window cover more source text). Without that control we'd have credited window expansion for gains that were actually just density.
On bzip2, pre-tokenization is a flat loss (about -3.9%), at every corpus size. bzip2's Burrows-Wheeler transform is exploiting byte-level structure in the prose that tokenization destroys, so it's simply the wrong backend to pair this with.
It is usually not a size-for-speed tradeoff. On 5 of the 7 backends tested, parmar is smaller and faster than compressing raw bytes, at every tier, simultaneously — because the compressor is handed ~45% fewer bytes, and the time saved compressing them outweighs the time spent tokenizing. The two exceptions are informative rather than just noise: zstd at its fastest level is quick enough that tokenization itself becomes the bottleneck above a couple hundred MB, and bzip2 loses on both size and speed for the structural reason above.
Multithreading interacts with this in a way we hadn't anticipated. xz -T splits input into independent blocks once the stream is at least twice the dictionary size, and speedup tracks the resulting block count almost 1:1 until you run out of cores. Because pre-tokenization shrinks the stream fed to xz, it also shrinks the block count at a fixed block size — on a 1 GB corpus, raw bytes got 8 blocks and a 7.5x speedup from threading, while the tokenized version got 5 blocks and only 5x. So parmar was quietly giving up close to a third of the available multithreaded speedup in exchange for its ratio win, in the regime where block count sits below the core count. Interestingly, zstd's multithreading doesn't cost any ratio at all — it doesn't reset its window between jobs the way xz's independent blocks do, which is a concrete reason to prefer zstd if you need both threading and no ratio penalty.
One negative result we were happy to get: we'd built a hand-rolled worker pool (manual_pool) on the theory it might beat tiktoken's built-in batch tokenization. It won in exactly 8 of 16 comparable configurations — chance — with swings from -6.8% to +44% and no pattern by thread count, backend, or corpus size. Digging in, tiktoken's batch encode already releases the GIL and parallelizes internally in Rust, so there's no headroom left for Python-side coordination to claw back, and the swings we saw were mostly measurement noise on a step that only takes 1-3 seconds. The honest conclusion: the extra complexity isn't worth it, and we kept the negative result in the writeup instead of quietly deleting the code path.
Where this is actually useful
Grounded in the measurements rather than vibes:
- Archiving large prose corpora with LZMA or high-level zstd — smaller archive, shorter compression run, no downside.
- Anything stuck on gzip/deflate — the +15% gain shows up at every corpus size, including small ones, because gzip's window is always saturated. If you can't swap the compressor but can change what feeds it, this is the cleanest win in the whole project.
- Text that's getting tokenized anyway — LLM training shards, eval sets, retrieval corpora — since a reader can skip re-tokenization on the way back out.
- Cold storage, write-once/read-rarely — decompression carries a detokenization cost that compression doesn't, so the asymmetry favors data you don't read often.
And where it explicitly isn't a fit: it's untested on non-prose (code, JSON, logs), it's not a general-purpose archive format (the archive stores the tokenizer's name, not its vocabulary, so you need the exact same tiktoken encoding available to decompress), and it has no encryption or authentication — the sha256 in the footer is an integrity check, not a MAC.
Try it
python -m venv venv
./venv/Scripts/python.exe -m pip install tiktoken numpy zstandard psutil matplotlib pandas
# smoke test
python matrix.py smoke --corpus ./corpus/pg19_64mb.txt
# or run the whole programme end to end
bash run_all.sh
Full results, the matrix-generation logic, and FINDINGS.md (a running list of everything that turned out to be wrong once the code could actually be run — including a fun one about the PG-19 dataset no longer being loadable the documented way) are all in the repo:
There's also an auto-generated code walkthrough on DeepWiki if you want to navigate the codebase rather than the results.
Open questions we'd like help on: sweeping dictionary size directly instead of corpus size (should isolate the plateau mechanism much more cheaply), frequency-remapping token IDs before packing to close the gap between LEB128 and the fixed-width packing on large-vocabulary tokenizers, and testing this on source code instead of prose. Issues and PRs welcome.
Top comments (0)