Field notes from a weekend spent teaching an entropy coder to speak LLM.
LLMs spent billions of dollars learning the best subword dictionary that has ever existed. Somewhere around 200,000 pieces of language, ranked by how much they compress. And I kept looking at that dictionary and thinking: nobody is allowed to just use that for compression?
TokPress is a pure-Python lossless compressor that tokenizes input with OpenAI's o200k_base vocabulary, applies token-level LZ77, and entropy-codes the result with rANS. For many small, schema-homogeneous records (logs, JSON, telemetry) a trained dictionary takes the ratio to 0.2565, and a batch mode reaches 0.0875x — while compressing each record alone can inflate it past 1.0. It beats gzip on prose and loses honestly to zstd's trained dictionary.
So I built one. Pure Python, no native code, one weekend, way too many cups of coffee. It's called TokPress, it compresses by tokenizing with o200k_base (the tokenizer OpenAI's models use), applying token-level LZ77, and entropy-coding the result with rANS. And along the way it produced a number that still makes me grin: 150 JSON log lines, compressed as one stream, down to 0.0875× their size — while compressing each one alone made them bigger.
This is the honest version of how that happened. Including the parts that broke.
Why "many small records" is a genuine pain
Normal compressors like gzip and zstd are built for big files. They learn their model from the stream, which is great when you have megabytes to amortize it over. But the world is full of small, independent, schema-similar records: JSON log lines, telemetry events, API responses, package metadata. Each one is 200 to 500 bytes. Each one is on its own. As an engineer who spends most days building ML systems in production — and who has spent weekends benchmarking tokenizers — this is the shape of half the data I touch.
Compress a single 200-byte log line with gzip and you'll find the header and the dictionary setup cost more than the content you saved. In my measurements, a lone 70-byte JSON sample came out at 81 bytes compressed — a compressor that made things bigger. That's not a bug; it's the cold-start penalty of not having a model.
Zstandard's answer is dictionary mode: train a shared dictionary offline, then every record compresses against it. MongoDB and Amazon DocumentDB now ship exactly this (up to 5× better ratio on JSON documents). And in 2025 the IETF shipped RFC 9842, which put dictionary compression into HTTP itself — browsers, servers, the works. "Known-structure API responses" is literally named as the sweet spot.
So: shared dictionary + many small homogeneous records = a real, money-sized problem. I just wanted to add one twist — use a tokenizer as the dictionary.
Beat 1 — the idea: the tokenizer is the free dictionary
Here's the pitch. A BPE tokenizer's whole job is finding the pieces of language that compress well. "action" is one token. "click" is one token. The tokenizer has already done the hard vocabulary work — for free, at huge scale.
So: tokenize the input, run LZ77 over the token ids instead of bytes, then entropy-code with rANS. The token stream is a much better-shaped alphabet than raw bytes, because the multi-byte structure is already baked in.
import tokpress
compressed = tokpress.compress(payload) # bytes or str
original = tokpress.decompress(compressed) # byte-exact
There's one non-negotiable requirement: byte-exactness, even for arbitrary binary. Logs aren't always valid UTF-8. tiktoken's public encode takes a str, but its internal _encode_bytes/decode_bytes pair operates on raw bytes — including a lone 0xFF. That's what I use. It round-trips anything.
Beat 2 — the first wrong turn (and it was a good one)
My first instinct was a domain vocabulary. Train on JSON logs, mine the frequent pieces, get a smaller tokenizer tailored to my data. Sounds great. It silently failed.
The problem: a restricted vocabulary is only a valid tokenizer if it forms a complete hierarchical BPE merge chain — every token has to be the concatenation of two lower-ranked tokens. Mining pieces by longest-match gives you a bag of strings, not a chain. The encoder can't reproduce the merges, so the "tokens" don't round-trip the way you think. I threw the whole domain-profile system out and learned the hard way:
A vocabulary you can't merge is a vocabulary you can't trust.
Beat 3 — three real bugs, all found by being paranoid
Building the entropy coder was where the weekend's real work was. rANS at table precision 2^16 with a 64-bit state, and a selector that builds up to eight candidate encodings per record and keeps the smallest. That last bit is a nice design: min(candidates, key=len) — nothing is ever forced, the size decides.
And it turned out the size decided wrong in three delightful ways. All three were caught only because I looped fuzz inputs over and over instead of testing one fixed example:
The single-symbol truncation. A table with exactly one active symbol gets frequency exactly 65536 — 100% probability. That needs 17 bits. I was writing it in 16, silently truncating 65536 to 0 on the wire. Fix: transmit
freq - 1.The reverse-order escape. rANS encodes in reverse logical order. A two-event cascade (context table → fall through to order-0) needs its encode calls issued in the opposite micro-order from how the decoder consumes them. Get it backwards and it corrupts output only on inputs that exercise the escape path — invisible until it isn't.
The double-reverse. In adaptive-split mode, the escape list is built in a forward pass before the reverse encode loop — unlike every other mode. So I reversed it a second time, by analogy. Wrong. Only when a record had more than one escape did the values come out scrambled.
Three bugs, three regression tests, zero drama. This is the part of compression that doesn't make blog headers but is 90% of the actual work: the wire format must be a perfect mirror between encode and decode, and the only way to trust it is to fuzz it to death.
Beat 4 — the dictionary, and the number that made it click
The real win came from a TokDict: train once on a sample of schema-homogeneous records, then every future record gets three free gifts — an LZ priming buffer (match against other records' history), a baked order-0 rANS table (no per-record table on the wire), and order-1 context tables for the most common previous-token contexts, with an escape cascade so a record can always contain something the dictionary never saw.
tokpress train-dict mydict.tokdict sample1.json sample2.json ...
tokpress compress new_record.json --dict mydict.tokdict -o new_record.tokz
tokpress decompress new_record.tokz --dict mydict.tokdict -o restored.json
On 230 held-out structured-log records (trained on 184, tested on 46 it never saw):
| Stage | ratio |
|---|---|
| per-record, no dictionary | 0.800 |
| + TokDict order-0 table | 0.284 |
| + order-1 context tables | 0.2565 |
| gzip -9 | 0.728 |
| zstd -19, no dictionary | 0.737 |
| zstd -19 + matched trained dict | 0.195 |
That's below gzip and dictionary-free zstd, and within 1.3× of zstd given the same training data — down from a 5–7× gap with no dictionary at all. Honest footnote: zstd's COVER/FastCover dictionary training is more mature than my concatenation-based priming, and it still wins on the same data. I'm not claiming otherwise.
And then the batch mode, which is the number I actually open the repo for. Instead of compressing each record on its own, concatenate them all and compress as one stream, with the record lengths stored in the header so decoding is still per-record exact:
packed = tokpress.compress_many(records)
records = tokpress.decompress_many(packed) # byte-exact, per-record
On 150 schema-homogeneous JSON records (~11.6 KB): compressing each separately summed to ratio 1.19 — the records literally grew. As one adaptive stream: 0.0875. Same bytes, one header instead of 150, one adaptive model spanning the batch instead of 150 cold starts. That's the whole thesis of the project in a single number.
Beat 5 — the turn: tokenization is not the magic
Here's the uncomfortable part, and it's a theorem: tokenization cannot reduce the information content of a message. It's injective, so it preserves entropy. Every compression gain has to come from modeling, never from the transform itself.
I verified this the hard way. In bulk mode, without a trained dictionary, TokPress is just an LZ + entropy pipeline — and on long prose it now beats gzip and, on the short-prose corpus, even zstd outright (0.298 vs 0.324, and brotli 0.306) — but that's the entropy coding doing the work, not the tokenizer. Take the tokenizer away and the wins shrink. The tokenizer's job is to reshape the alphabet so a low-order model sees structure that would need a high-order byte model. It's a lever, not a free lunch.
This also tells you where TokPress sits next to the closest thing it has to a cousin. The parmar project measured "tokenize before you compress" across 452 configurations and found tiktoken-then-xz beats plain xz by 7–9.6% — but it pipes the token IDs into a byte-level compressor, and it explicitly left code, JSON, and logs untested. TokPress goes the other way: it entropy-codes the token IDs directly with rANS, and it's built for exactly those many-small-records, schema-homogeneous corpora parmar skipped. Same starting bet, opposite half of the design space — and the honest result is that on whole-file prose the direct route wins over piping tokens through xz, while neither is a substitute for a trained dictionary.
The other honest finding: a trained dictionary is schema-specific. Train on JSON logs, apply to Python code, and the dictionary buys you nothing over no dictionary at all (0.489 vs 0.485). It's a genuinely domain-specific artifact. Use it on data that resembles its training data, and only then.
Along the way I also learned the machine's own dirty secret: tiktoken's _encode_bytes routes valid UTF-8 through a regex (pat_str) and BPEs per piece — so a vocabulary trained with naive whole-input BPE fragments on piece boundaries. The fix was training the same way tiktoken encodes: pre-tokenize with the same regex, forbid merges across pieces. Once I did that, a JSON-trained vocabulary beat o200k_base on held-out JSON (0.162 vs 0.178). The project now ships tokpress train-vocab and tokpress fit (vocab + dictionary in one shot).
The toolbox, one week later
The weekend project grew a full CLI while I wasn't looking:
tokpress compress record.json -o record.tokz
tokpress decompress record.tokz -o restored.json
tokpress pack batch.tokz record1.json record2.json ... # one adaptive stream
tokpress unpack batch.tokz out_dir/
tokpress read batch.tokz 7 # O(1) random access
tokpress train-dict mydict.tokdict samples.jsonl
tokpress train-vocab myvocab.ranks corpus.txt
tokpress fit out corpus.txt # both at once
tokpress tokenize-stats file.txt # tokens/KB, entropy, MI
That last one is a little love letter to the LLM crowd: it reports the order-0/order-1 token entropy and adjacent-token mutual information of any corpus — compression as a tokenizer-quality signal, which Goldman et al. showed correlates with actual model performance. Tokenize your eval set, get a number that says something about how well the tokenizer fits it.
The whole thing is deliberately un-optimized, pure Python, ~2900 lines, 93 tests. Decompression is fast (thousands of records/sec); compression is slower (the encoder builds every candidate mode and keeps the smallest — thorough, not fast). I made the SymbolStats alphabet pass active-symbol-only one afternoon and got a 2.8× speedup with byte-identical output, which was the most satisfying 40 lines of the weekend.
Share your thoughts in the comments — I’d love to hear how this technology is impacting your industry.
👉 Be sure to press the like button and follow me. It would be a great motivation for me.
👉 Follow me: LinkedIn | GitHub
The takeaway
A tokenizer is a compressor's dictionary that someone else already trained. Reusing
o200k_baseas the alphabet is free leverage — just don't expect the tokenizer alone to be the win.Many-small-homogeneous-records is its own regime. One stream over 150 records: 0.0875×. One record at a time: 1.19×. The container is the feature.
Modeling is everything; the transform is a lever. Tokenization preserves entropy — every gain is the entropy coder and the trained dictionary doing real prediction work.
Honest numbers beat impressive ones. It beats gzip on prose, ties/beats zstd on some corpora, and still loses to zstd's trained dictionary by 1.3×. I measured all of it, and the losing parts are the parts that made me learn the most.
If you want to poke at it, it's all open source: github.com/LakoreAI/tokpress. The bench harness (scripts/bench.py) is the source of truth for every number in this post — it round-trip-checks every ratio it prints. I'd love to see what it does on your logs.
References
The papers I kept going back to while building this — the entropy coder, the tokenizer, and the "compression is prediction" framing that got me started.
J. Duda, Asymmetric Numeral Systems, arXiv:0902.0271. The rANS coder — the entropy stage is a direct implementation of the range variant.
J. Ziv and A. Lempel, Compression of Individual Sequences via Variable-Rate Coding, IEEE Trans. Inf. Theory, 24(5), 1978. Token-level LZ77, plus the finite-length redundancy that makes small records the interesting regime.
P. Gage, A New Algorithm for Data Compression, The C Users Journal, 12(2), 1994. Byte-pair encoding — the ancestor of every subword tokenizer here.
A. Radford, J. Wu, et al., Language Models are Unsupervised Multitask Learners, OpenAI, 2019. The byte-level BPE tokenization that became
o200k_base.T. Kudo and J. Richardson, SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, EMNLP (demo), arXiv:1808.06226, 2018. The subword-modeling framing behind the whole tokenizer-as-dictionary idea.
G. Delétang, A. Ruoss, et al., Language Modeling Is Compression, arXiv:2309.10668, 2023. The prediction-compression equivalence — and the honest "no free lunch" reminder that the transform never carries the win.
C. S. K. Valmeekam, K. Narayanan, D. Kalathil, J.-F. Chamberland, S. Shakkottai, LLMZip: Lossless Text Compression using Large Language Models, arXiv:2306.04050, 2023. The LLM-as-predictor end of the spectrum; TokPress is the cheap, static-dictionary end of the same idea.
O. Goldman, A. Caciularu, M. Eyal, K. Cao, I. Szpektor, R. Tsarfaty, Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance, arXiv:2403.06265 (EMNLP Findings), 2024. The result behind
tokpress tokenize-stats— tokenizer compression as a quality signal.Y. Collet and M. Kucherawy, Zstandard Compression and the application/zstd Media Type, RFC 8478, 2018. The main comparison point, including its dictionary mode I keep losing to.
J. G. Cleary and I. H. Witten, Data Compression Using Adaptive Coding and Partial String Matching, IEEE Trans. Commun., 32(4), 1984. PPM — the escape-to-lower-order idea the per-record order-1 mode borrows.
Originally published on ZyVOP
💡 For more articles like this, subscribe to the ZyVOP newsletter!
Top comments (0)