JTOKEN - Lossless JSON compression for LLM prompts

Same data, ~35% fewer tokens. Purpose-built for RAG pipelines, AI agents, and structured prompt engineering.

Hey Folks 👋

I'm Hermann Samimi, a data engineer who's been building RAG pipelines professionally. At some point I started looking at what my prompts were actually made of, and noticed something embarrassing: a huge chunk of tokens were going to JSON syntax — braces, quotes, commas, true, false, null — not the actual data I cared about.

So I built jtoken.

It encodes JSON into a flat, human-readable format optimized for LLM context windows. The encoding is lossless — you can always round-trip back to the original. It works on plain JSON, but also natively handles MongoDB documents (shell and extended JSON) and Elasticsearch hits, which is where I personally saw the biggest savings.

Real numbers:
→ Elasticsearch hit: 1,537 → 583 tokens (62% reduction)
→ MongoDB shell doc: 770 → 508 tokens (34% reduction)
→ Standard JSON: 617 → 503 tokens (18% reduction)

The format is simple enough to understand in 30 seconds:

Input:
{"name": "Alice", "active": true, "verified": false, "ref": null}

Output:
name: Alice
trues: active
falses: verified
nulls: ref

There's also a CLI for quick stats and a token counting API that works with tiktoken or a heuristic fallback.

pip install jtoken

I'd love feedback — especially on edge cases, format ideas, or use cases I haven't thought of yet. What do you use to pass structured data into LLMs today?

PyPI: https://pypi.org/project/jtoken/
GitHub: https://github.com/HermannSamimi/jtoken
Linkedin: https://www.linkedin.com/in/hermann-samimi/