DEV Community

Rudson Kiyoshi Souza Carvalho
Rudson Kiyoshi Souza Carvalho

Posted on

TERSE — A New Serialization Format Built for LLMs

#llm #opensource #ai #token

JSON is the default. But defaults were built for a different world.

Every time you send structured data to a Large Language Model, you pay for it token by token. And if you're using JSON — which almost everyone is — you're paying for a lot of characters that carry no information.

Take this simple payload:

{
  "user_id": 1001,
  "status": "active",
  "data": ["feature_a", "feature_b"],
  "verified": true
}
Enter fullscreen mode Exit fullscreen mode

Count the noise: braces, quotes around every key and string value, commas, colons with spaces. Now imagine this multiplied across thousands of API calls per day. That's real money.

I built TERSE to address this.

What is TERSE?

TERSE (Token-Efficient Recursive Serialization Encoding) is a text-based data serialization format designed to represent the complete JSON data model with substantially fewer tokens — making it significantly more cost-efficient for use as input to Large Language Models.

The same payload in TERSE:

user_id: 1001
status: active
data: [feature_a feature_b]
verified: T
Enter fullscreen mode Exit fullscreen mode

Same information. ~47% fewer tokens.

How it compares

Format Token savings vs JSON Full JSON coverage?
JSON baseline ✓
YAML ~20% ✓ (verbose arrays)
TOON ~40% ✗ (flat data only)
TERSE ~47% ✓

YAML is a genuine improvement over JSON — it's more compact and covers the full data model. But it was designed for humans to write, not for LLMs to consume. Verbose arrays (- item per line), full-word booleans (true/false), and a notoriously complex parser spec limit its token savings.

TOON goes further on token reduction but falls apart with nested objects — it only works for flat, uniform tabular data. If your payload has any nesting, TOON can't represent it.

TERSE was designed to close that gap: full JSON data model coverage, with token efficiency as the primary design constraint.

The five design principles

1. Bare strings — identifiers and common values require no quotation marks. production stays production, not "production". Quotes are reserved for strings that actually need them — those containing spaces, reserved characters, or special syntax.

2. Compact primitives — null, true, and false become single characters: ~, T, F. Three of the most common values in any payload, each reduced to one token.

3. Implicit delimiters — spaces separate values inside objects and arrays. No trailing commas, no colons between array elements.

4. Schema arrays — the biggest token win for tabular data. Uniform arrays of objects declare their fields once, then list values positionally:

users:
  #[id name role active]
  1 Alice admin T
  2 Bruno editor T
  3 Carla viewer F
Enter fullscreen mode Exit fullscreen mode

The equivalent JSON repeats "id", "name", "role", "active" on every single row. For a 100-row dataset, that's 400 unnecessary key repetitions.

5. Recursive structure — all constructs nest arbitrarily. Objects inside arrays inside schema arrays — all valid, all compact. No flat-only limitations.

A real example: nested order

JSON (~180 tokens):

{
  "orderId": "ORD-001",
  "customer": {
    "name": "Rafael Torres",
    "email": "r@email.com"
  },
  "items": [
    {"sku": "A1", "qty": 2, "price": 9.99},
    {"sku": "B3", "qty": 1, "price": 24.50}
  ],
  "paid": true,
  "notes": null
}
Enter fullscreen mode Exit fullscreen mode

TERSE (~95 tokens):

orderId: ORD-001
customer: {name:"Rafael Torres" email:r@email.com}
items:
  #[sku qty price]
  A1 2 9.99
  B3 1 24.50
paid: T
notes: ~
Enter fullscreen mode Exit fullscreen mode

This is where TERSE separates itself from TOON and CSV — deeply nested structures work exactly as expected.

You don't write TERSE by hand

The workflow is identical to JSON:

Your data (object/dict)
      ↓
serialize()        ← terse-js or terse-py
      ↓
TERSE string       ← sent to the LLM
      ↓
parse()            ← if you need it back
      ↓
Your data again
Enter fullscreen mode Exit fullscreen mode

Just like nobody writes JSON.stringify() output by hand — you call the function. TERSE works the same way. The format is optimized for the one reader that actually matters: the LLM.

On design intent: why not compress further?

TERSE could go deeper — automatic key abbreviation, binary type encoding, dictionary compression. We deliberately stopped short of that.

The goal is a format that remains human-auditable: you can open a .terse file in any text editor and understand what you're looking at without tooling. In LLM pipelines, auditability is a safety property, not just a convenience. When an agent misbehaves, you need to inspect its inputs.

Two questions that come up

Can I use TERSE for REST API communication between microservices?

You can, but it's not the primary use case. REST APIs are consumed by many clients across different teams and languages — JSON's universal support is a real advantage there. TERSE shines where you control both ends: serializing data before sending it to an LLM, and parsing the response on the other side.

Can I use TERSE for application configuration, like YAML?

Yes — the format supports everything YAML does for config files: nested objects, arrays, typed values, comments. Worth considering if your config is also consumed by an LLM as context.

What's available today

The project includes:

  • Formal specification (v0.7) with ABNF grammar, conformance rules, and security considerations — published on Zenodo with DOI: 10.5281/zenodo.19058364
  • Reference implementations in TypeScript, Python, Java, and Go
  • Live playground where you can paste JSON and see the TERSE output in real time

Everything is open source under MIT (implementations) and CC BY 4.0 (specification).

Links

TERSE is still a draft — v0.7 is open for community review. If you work with LLM pipelines at scale, I'd love to hear whether this addresses a real pain point in your stack.

Rudson Kiyoshi Souza Carvalho — Independent Researcher

Top comments (0)