DEV Community

Cover image for A Simple Look at TokPress: A Compressor That Uses an LLM's Tokenizer
firefrog
firefrog

Posted on Originally published at zyvop.com

A Simple Look at TokPress: A Compressor That Uses an LLM's Tokenizer

If you've ever compressed a bunch of small JSON log lines, you've probably noticed something annoying: gzip often makes them bigger, not smaller. A 200-byte log line has so little content that the compressor spends more bytes on its own bookkeeping than it saves.

TokPress is a tiny weekend project that tries a different trick: instead of compressing raw bytes, it first runs the text through the same tokenizer that OpenAI's models use (o200k_base), then compresses the tokens.

That's the whole idea. An LLM tokenizer has already learned which pieces of text are common — words, punctuation, code patterns. So the compressor gets a much better alphabet to work with for free.

What it actually does

Three simple steps:

  1. Tokenize — turn the input into token ids using o200k_base.

  2. Compress the tokens — run a simple LZ77 pass over the token ids.

  3. Entropy-code — finish with rANS, an efficient entropy coder.

The result is a .tokz file that decompresses back to the exact original bytes — even binary data, since the tokenizer works on bytes, not just text.

import tokpress

compressed = tokpress.compress(payload)
original = tokpress.decompress(compressed)   # byte-exact
Enter fullscreen mode Exit fullscreen mode

The trick that makes it useful: a shared dictionary

The real win comes when you have many records that look the same — log lines, API responses, telemetry. TokPress can train a small dictionary on a sample of your records, then every future record compresses against it:

tokpress train-dict mydict.tokdict samples.jsonl
tokpress compress new_record.json --dict mydict.tokdict -o new_record.tokz
Enter fullscreen mode Exit fullscreen mode

With a trained dictionary, structured-log records went from a ratio of 0.800 to 0.2565 — about 3× smaller. And that's on records the dictionary had never seen.

There's an even simpler mode that needs no training at all: compress many records together as one stream, and the model learns as it goes:

packed = tokpress.compress_many(records)
records = tokpress.decompress_many(packed)
Enter fullscreen mode Exit fullscreen mode

On 150 schema-similar JSON records, per-record compression summed to ratio 1.19 — the data literally grew. As one stream, it hit 0.0875. Same bytes, one header instead of 150.

Where it stands (honestly)

  • On prose it beats gzip and ties/beats zstd on some files (a 152KB prose file: 0.298 vs zstd's 0.324).

  • It still loses to zstd's trained dictionary by about 1.3×. zstd has had years of polish on dictionary training; this is a weekend project.

  • It's pure Python, so it's slow to compress. Decompression is fast (thousands of records/sec); compression is not.

  • Tokenization isn't magic. The tokenizer just reshapes the data; the real savings come from the entropy coding and the trained dictionary.

Try it 

git clone https://github.com/LakoreAI/tokpress
cd tokpress && pip install -e .
python -c "import tokpress; print(len(tokpress.compress(b'{\"a\": 1}')))"
Enter fullscreen mode Exit fullscreen mode

It's ~2900 lines, has 93 tests, and comes with an honest benchmark script. If you have a folder of repetitive logs or JSON, that's exactly the case it was built for.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Top comments (0)