Token Optimization for LLMs: What I Learned Testing Alternatives to JSON

I Was Paying for the Same Word 300 Times and Didn't Even Know It

---

What I learned digging into token optimization and data strategy for LLMs — and why the format you send data in matters more than I thought

---

A few weeks ago I noticed something dumb.

I was sending a batch of records to an LLM, formatted as JSON like always. And staring at the payload, I realized the same field name was showing up over and over. Once per row. Three hundred rows meant the same word repeated three hundred times, telling the model nothing it didn't already know after row one.

That bugged me more than it probably should have. So I went looking for an answer to a pretty basic question: is there a better way to send data to an LLM, or is JSON just what we all use because it's what we've always used?

That question turned into a proper rabbit hole. This is what I found.

Why I even started caring about this

I'll be honest, I didn't wake up thinking about token costs. I was building a pipeline that sends structured data into prompts, and I kept watching the token count climb every time the data got bigger. It felt wasteful. I just didn't have a name for why it felt wasteful yet.

Why tokens matter in the first place

Here's the part that clicked for me. LLMs don't read text the way we do. They break everything into tokens, rough chunks of words or sub-words, and every single token costs money and eats into the context window.

Now think about a JSON array of a few hundred rows. Every row repeats the same field names. "id", "name", "status", over and over, identical every time. None of that repetition is teaching the model anything new. It already understood the shape of the data from the very first row.

So what is all that repeated text actually doing? Nothing useful. It's just padding.

Cutting that padding doesn't change what data the model sees. It only removes the noise around it. Same information, fewer tokens. That distinction, padding versus actual information, is the whole reason this rabbit hole exists.

The three formats, and what they're each actually for

Once I understood the problem, I expected there'd be one obvious "correct" format to replace JSON with.

There isn't. There are at least three, and each one is answering a slightly different question.

JSON

JSON won because it's universal. Every language parses it, every API speaks it, every database driver serializes to it. That's not going away, and honestly, it shouldn't.

The problem isn't JSON itself. The problem is specifically what happens when JSON goes into an LLM prompt. Its self-describing structure, repeating "key": on every single object, is exactly the kind of redundancy that costs money at scale without adding any value for the model reading it.

Keep using JSON for: APIs, databases, config files. Anything that isn't going straight into a prompt.

A token-efficient tabular format

The first alternative I looked at is built specifically for flat, tabular data, basically the shape of a spreadsheet or a database query result.

Instead of repeating field names on every row, it declares the fields once in a header, then lists nothing but values underneath:

items[4]{id,name,category,status}:
IT-001,Widget-A,tools,active
IT-002,Widget-B,parts,inactive
IT-003,Widget-C,tools,active
IT-004,Widget-D,parts,active

That's it. No repeated keys. No repeated brackets or quotes on every line. Just a header, once, then rows of values.

I almost skimmed past one small detail in that header the first time I saw it, and it turned out to matter a lot later. See that [4]? That's a declared row count. If a payload gets cut off somewhere in a pipeline, network hiccup, buffer limit, a bug in a serialization step, whatever, the model can compare "I was told 4 rows" against "I only actually received 3." JSON has nothing like that built in. There's no field anywhere that says "expect this many entries," so a truncated JSON array just quietly looks like a shorter array. Nothing about the format itself raises a flag.

Best for: product catalogs, log lines, database exports, search results, anything that's naturally a table.

A format built for nested structure

The second alternative targets a completely different shape of problem: nested, repeated structure. Think file trees, multi-step agent messages, API responses where the same object shows up in five different places inside the same payload.

Instead of copying a repeated structure out in full every time it appears, this format defines it once and then references it by an ID wherever it shows up again:

{
  "items": {
    "@id": "item-1",
    "name": "Widget-A",
    "children": [
      { "@ref": "item-1" }
    ]
  }
}

Rather than pasting the full object again every time it recurs, you just point back to it. One definition, many references.

Best for: multi-tool agent workflows, nested API responses, anything where the same data structure shows up inside itself more than once.

So which one should you use?

Here's the part that took me a minute to actually get straight in my head. These aren't three competing options where you pick the "best" one and move on. They solve different problems.

The tabular format is about flat rows. The nested-structure format is about repeated hierarchy. JSON is about universal compatibility outside the prompt entirely. Which one is right depends on the shape of your data, not on which format has the flashier claims on its documentation page.

Schema definitions are a separate decision entirely

This is the part I got wrong at first. I assumed "how do I send data efficiently" and "how do I tell the model what to send back" were the same question with the same answer.

They're not.

Data formats govern what you send to the model. Schemas govern what you ask the model to send back. Different problem, different answer.

For backend validation, things like API contracts or database constraints, a verbose schema format is genuinely the right tool. It spells out "type": "object", "properties": {...} in exhausting detail, and that verbosity is the entire point. It's unambiguous, and code can check against it automatically.

But that same verbosity is dead weight inside a system prompt. Every token spent on schema boilerplate is a token not spent on actual instructions.

The fix I landed on: describe the output shape using something closer to a type definition instead.

interface Item {
  id: string;
  name: string;
  status: "active" | "inactive";
}

Same structural guarantee. A fraction of the tokens. LLMs have seen an enormous amount of this kind of syntax during training, so they follow it reliably without needing the schema spelled out the long way.

Rule of thumb I've been using since: verbose schema formats for code that validates. Lightweight type-style definitions for prompts that generate.

Okay, but does any of this actually work?

At this point I had a working theory: swap the format right before it hits the prompt, keep JSON everywhere else, and save a meaningful chunk of tokens with no real downside.

I didn't want to just trust that theory. Claims about token savings are everywhere, and most of them come from the people selling the format. So I ran an actual comparison instead of taking anyone's word for it.

The setup, roughly:

• A flat dataset of general-purpose records, run through JSON, the tabular format, and the nested-structure format
• One model, called through a standard routing setup
• A fixed set of questions per format, covering field retrieval, aggregation, filtering, structure awareness, and one specifically designed corruption check, run multiple times each
• Grading done against ground truth computed directly from the data, not graded by another LLM Worth being upfront: this was one model, one dataset, one run. I'm treating what follows as a directional signal, not a universal verdict on any format.

What I actually found

Token and byte savings. The two alternative formats landed almost dead even with each other, both cutting token usage to roughly 45% of what JSON used, a reduction of about 55%. One of the two formats' own documentation claims a meaningful edge over the other. That edge didn't show up in my run. They were within a point of each other on both tokens and bytes.

One thing kept me honest here: plain gzip-compressed JSON alone gets to a much bigger byte reduction than either format managed. A model can't read gzipped text, so that's not really a competing option, but it's a useful reference point. It means a real chunk of the savings from either format is simply "remove JSON's punctuation and repeated keys," not something uniquely tied to how an LLM specifically processes text. The token-level win is still real. It's just not entirely magic.

Accuracy. This is where I expected the leaner formats to have some kind of edge, since they're theoretically handing the model a cleaner signal with less noise to parse through. That's mostly not what happened. Accuracy was close across all three formats, and none of them was a clear winner. One of the two alternative formats actually finished slightly behind plain JSON.

That matters more than it might look at first glance. Cutting tokens isn't something to do blindly if it comes with an accuracy cost, and this run doesn't rule that out.

One pattern showed up no matter the format: questions asking the model to count or filter across the entire dataset scored badly everywhere. The model seems to estimate rather than exhaustively count once a list gets long enough, and no data format fixes that. It's a limit of how the model reasons over long lists, not something compression can paper over.

Latency. Response times varied a bit between formats, but not in any consistent direction worth reading into from a single run.

The part that actually surprised me

I went in expecting the token savings number to be the headline. It wasn't.

I deliberately truncated each payload by dropping the last few rows, without touching the declared row count, and asked the model whether the data looked complete.

Every format got the model to say "this looks incomplete." Every single time. On a simple pass/fail basis, that's a three-way tie.

But why the model said that was completely different depending on the format.

For the two alternative formats, the header still declared the original row count while fewer rows were actually present. The model could point to that exact mismatch as evidence. A real, checkable fact, not a guess.

For JSON, there's no count field anywhere to check against. The model still said "looks incomplete," but the reasoning behind that answer read more like a vague hunch about typical dataset sizes. A plausible-sounding guess dressed up as an answer, not an actual check.

If I'd only looked at the raw pass/fail number, I would have missed this completely. All three scored identically. It was only after separating "said incomplete" from "said incomplete and cited the actual mismatch as evidence" that the real gap showed up. Both alternative formats hit a perfect score on that stricter version. JSON hit zero.

This is, to me, the most useful finding in the entire investigation, and it's barely about token count at all. It's about giving the model something structural to actually check its work against, in any pipeline where payloads can get silently truncated somewhere upstream.

What this means if you're building with LLMs

A few things I'm taking away from this.

The format question isn't "which one is best." It's "what shape is my data." Flat and tabular points one direction. Deeply nested and repeating points another. That decides the format before token count even enters the conversation.

Fewer tokens doesn't automatically mean better answers. One alternative format edged out JSON on accuracy in this run. The other came in slightly behind it.

Test the truncation case specifically before rolling any of this out for real, not just the token count. The gap between "sounds right" and "is actually grounded in something checkable" only showed up because I checked for it directly. The surface-level pass rate looked identical across every format.

And JSON isn't going anywhere, nor should it. It stays the standard for APIs, databases, and config files. This whole thing is about one specific moment, right before a payload goes into a prompt, and nowhere else.

One smaller win worth mentioning: for schema definitions, moving away from verbose validation-style schemas and toward lightweight type-style definitions inside prompts is close to free. Same guarantee, a fraction of the tokens.

What I actually learned

I started this because of a repeated field name that annoyed me more than it should have. I ended up with a genuinely different mental model for how I think about sending data to an LLM.

The token savings were real, roughly half, and they held up under an actual test instead of just a claim on a docs page. That part confirmed what I suspected going in.

What I didn't expect was that the more interesting question turned out to be something else entirely: not "how many tokens does this save," but "does this give the model something concrete to check itself against." That distinction barely shows up in most of the marketing around token optimization, and it's the one I keep coming back to.

If there's one thing worth remembering out of all this: a smaller payload is good. A payload the model can actually verify is better. Those aren't automatically the same thing, and it took an actual benchmark for me to see the difference.