DEV Community: İsmail Kağan Acar

TOON Benchmarks: A Critical Analysis of Different Results

İsmail Kağan Acar — Fri, 07 Nov 2025 10:07:10 +0000

TOON (Token-Oriented Object Notation), a new data format is making brave claims. It says it can cut down on tokens by 30–60% compared to JSON, while also helping LLMs understand the data more accurately.[1]

However, when you start looking at the test results (the benchmarks), you see a difference between the official scores and the results from independent testing.

So, this article is going to lay out both sets of data, side-by-side, and look into what might explain why they don’t match up.

Understanding the Benchmark Methodology

Before we dive into the results, it’s important to know what these tests actually measure.

They check how well an LLM can understand and pull information from data. Specifically, we’re seeing how good it is at answering questions about data when it’s presented in different formats. [1]

So, think of it as a reading test, not a test of whether the model can write in that format.

Official Repository Benchmarks

Headline Results

The official TOON repository presents compelling performance metrics based on 201 data retrieval questions across 4 models:[1]

Key Finding: TOON achieves 68.7% accuracy versus JSON’s 65.7% while using 39.5% fewer tokens — a seemingly definitive win for TOON on both efficiency and accuracy metrics.[1]

Dataset Composition

The official tests used five different types of datasets to see how the models performed:[1]

Tabular: 100 employee records, which were all simple and had the same fields.
Nested: 50 e-commerce orders, which were complex and had data tucked inside other data (like a customer object with a list of items).
Analytics: 60 days of time-series data (like dates and numbers).
GitHub: Real-world data from 100 popular GitHub repositories.
Event Logs: 75 logs, about half of which were simple and the other half had more complex error details.

Across all these datasets, the models were asked a total of 201 questions to retrieve information.

Model-Specific Performance

A separate benchmark study using the TOON implementation tested individual model performance, showing varying results across different LLMs:[1]

Here is your list, corrected using the official values you provided. I’ve included the full official results for each model for completeness and updated the rankings.

GPT-5-nano

TOON: 88.6% accuracy (178/201) — Best overall performer
JSON compact: 88.1% accuracy (177/201)
CSV: 88.0% accuracy (88/100)
YAML: 84.6% accuracy (170/201)
XML: 81.6% accuracy (164/201)
JSON: 80.1% accuracy (161/201)

Claude Haiku 4.5

YAML: 52.2% accuracy (105/201) — Best performer for this model
TOON: 50.7% accuracy (102/201)
JSON: 50.2% accuracy (101/201)
JSON compact: 49.8% accuracy (100/201)
XML: 49.3% accuracy (99/201)
CSV: 39.0% accuracy (39/100)

Gemini 2.5 Flash

XML: 86.1% accuracy (173/201) — Best performer for this model
TOON: 84.1% accuracy (169/201) — Ranked 2nd
CSV: 82.0% accuracy (82/100)
JSON compact: 81.1% accuracy (163/201)
YAML: 81.1% accuracy (163/201)
JSON: 81.1% accuracy (163/201)

Grok-4-fast-non-reasoning

TOON: 51.2% accuracy (103/201) — Tied for 1st
JSON: 51.2% accuracy (103/201) — Tied for 1st
XML: 50.2% accuracy (101/201)
JSON compact: 49.8% accuracy (100/201)
YAML: 48.8% accuracy (98/201)
CSV: 40.0% accuracy (40/100)

Note: These results are from a separate 201-question benchmark and may use different datasets than the main official repository’s 201-question benchmark.[1]

Independent Third-Party Benchmarks

Test 1: Tabular Data with GPT-4.1-nano

An independent evaluation from improvingagents.com presents dramatically different findings. Testing was conducted using GPT-4.1-nano across 12 formats with statistical confidence intervals:[2]

Key Finding: In this evaluation, TOON ranks 9th out of 12 formats with 47.5% accuracy, performing worse than JSON (52.3%) and significantly behind Markdown-KV (60.7%).[2]

The Accuracy-Token Efficiency Trade-off

This independent test shows that while TOON is great at saving tokens (using only 21,518 compared to 66,396 for JSON), this efficiency might come at a price.[2]

It seems that being too compact might make the data harder for the LLM to understand.

Here’s the interesting part: the format that got the best accuracy score (Markdown-KV at 60.7%) was actually a lot “heavier,” using more than double the tokens that TOON did.

This suggests that using more words and a clearer structure (being more “verbose”) might actually give the LLM the context it needs to understand the data better.

Test 2: Nested Data with GPT-5-nano

The same independent researchers conducted a second evaluation using GPT-5-nano to test TOON’s performance with nested data structures:[3]

Key Finding: In this nested data test, TOON ranked last with 43.1% accuracy, performing worse than JSON (50.3%), Markdown (54.3%), and significantly behind YAML (62.1%).[3]

This second test reveals a critical limitation: TOON’s performance degrades substantially with nested data structures, despite still being relatively token-efficient.[3]

Analyzing the Differences

The conflicting results present a puzzle: how can TOON score 68.7% in official benchmarks but only 47.5% in independent testing? Several factors warrant consideration:

1. Dataset Characteristics

The official tests and the independent tests were not testing the same things.

The Official Tests:

The creators of TOON set up their benchmarks using six different types of data. This included “tabular” (flat) employee records, which they openly state is the perfect, ideal use case for TOON.[1]

They basically created different test “tracks” (like flat data vs. mixed-structure data) that were heavily weighted toward scenarios where TOON was designed to win.

The Independent Tests:

The independent researchers focused only on tabular (flat) data and used one specific model, GPT-4.1-nano (which is a small, fast model).

Their write-up suggests that TOON “could be a format to consider if you’re trying to reduce token usage, especially if the limitations of the CSV format make it hard to represent aspects of your data.”[2]

This implies their tests might have used data or asked questions in a way that didn’t play to TOON’s biggest strengths, giving a different perspective on its performance.

2. Question Complexity

The distribution of question types matters significantly:

Simple field retrieval favors compact formats like TOON
Complex aggregation may benefit from more explicit structure
Nested queries could suffer in TOON’s flattened representation

The independent study doesn’t specify question distribution details, making direct comparison difficult.

3. Model Selection

This represents a critical methodological difference:

Official benchmarks: Tested across 4 different LLM models, providing an aggregate view of TOON’s performance.[1]

Independent benchmarks: Conducted two separate tests — one with GPT-4.1-nano on tabular data and another with GPT-5-nano on nested data.[2][3]

The model-specific results from separate testing show dramatic variance:

GPT-5-nano achieved 96.1% accuracy with TOON[1]
Claude Haiku 4.5 scored only 48.7% with TOON[1]
GPT-4.1-nano in the independent study: 47.5% with TOON[2]

Interestingly, the independent tests showed that even GPT-5-nano achieved only 43.1% accuracy with TOON on nested data[3], despite the official benchmarks showing GPT-5-nano reaching 96.1% with TOON.[1] This dramatic variance suggests the data structure type (tabular vs. nested) may be more significant than model selection alone.

4. Token Counting Methodology

Official benchmarks used o200k_base encoding (GPT-4o/GPT-5 tokenizer) via gpt-tokenizer.[1]

Independent benchmarks don’t specify their tokenization method.[2]

Different tokenizers can produce varying token counts for identical text, particularly with formatting characters and whitespace. This could affect both absolute counts and relative efficiency rankings.

Model-Specific Considerations

The official benchmarks reveal that TOON’s performance varies dramatically by model:[1]

Excellent: GPT-5-nano (88.6% on official benchmarks)
Good: Gemini 2.5 Flash (86.4%)
Moderate: Claude Haiku 4.5 (48.7%), Grok-4-fast (49.4%)

However, the independent nested data test showed GPT-5-nano achieving only 43.1% accuracy with TOON.[3] This suggests the data structure type matters more than model selection alone.

This variance is critical for practical applications. TOON’s performance depends heavily on both the model you’re using AND the type of data you’re working with. Even top-performing models like GPT-5-nano may struggle with TOON when dealing with nested structures.

Conclusion: An Evidence-Based Perspective

The available benchmarks paint an incomplete picture. TOON demonstrably reduces token usage by 30–60% across multiple studies — this finding appears robust.[1][2] However, accuracy results span from strong performance (68.7%) to below-average (47.5%), depending on the evaluation methodology.

This variance isn’t necessarily damning. It reflects the reality that no data format is optimal for all scenarios. TOON appears well-suited for:

Uniform tabular data
Simple field retrieval queries
GPT-5 and similar models
Applications where token efficiency is paramount

It may under-perform with:

Complex nested structures (where it ranked last in independent testing)
Deep reasoning queries
Claude Haiku and similar models
Applications where accuracy cannot be compromised

So, what’s the takeaway?

Here’s my recommendation: Treat TOON like a specialized tool, not a one-size-fits-all replacement for JSON.

If you’re thinking about using it for a real project, you have to test it yourself — with your data and your models — before you commit. The token savings look great, but you need to prove they don’t hurt the accuracy of the answers you need.

As TOON gets older and more independent researchers test it, we’ll all get a better idea of where it really shines and where older formats are still better.

Until then, let your own results guide you, not the hype.

Saving tokens is a big deal, and optimizations are important. But good optimization means you have to be clear-eyed about the pros and cons, not just accept the headline numbers.

TOON is an interesting experiment. It just needs a lot more attention, ongoing testing as the technology and our use cases for it continue to evolve.

References

[1]: Johann Schopplich, “TOON — Token-Oriented Object Notation”, GitHub Repository, https://github.com/johannschopplich/toon

[2]: “Is TOON Good for Table Data?”, Improving Agents, https://www.improvingagents.com/blog/is-toon-good-for-table-data

[3]: “TOON Benchmarks”, Improving Agents, https://www.improvingagents.com/blog/toon-benchmarks

TOON: New data format for LLM Applications

İsmail Kağan Acar — Fri, 07 Nov 2025 06:38:28 +0000

Understanding the Token Economy Problem

Large Language Models (LLMs) work by “reading” information in small pieces called tokens.

Think of tokens like building blocks. These tokens are important because they control two main things:

Cost: How much you have to pay to API’s or GPU’s.
Memory: How much information the LLM can handle at one time (this is its “context window”).

When you give the LLM organized data like Tool/API outputs, the format you use (like JSON) changes how many tokens it takes up.

Common formats like JSON are widely used but aren’t token-efficient. They often use more tokens than necessary, which drives up costs and uses up the LLM’s limited memory.

This is where Token-Oriented Object Notation (TOON) enters the field. Created in 2025, TOON is a compact data serialization format specifically designed for passing structured data to LLMs claiming 30–60% token reduction compared to JSON.[1]

What is TOON?

TOON is a text-based data format that prioritizes token efficiency while maintaining human readability. Unlike JSON, which repeats field names for every object, TOON uses a tabular approach where field names are declared once in a header, and subsequent rows contain only values.[1]

The Core Syntax

TOON combines familiar patterns from YAML (indentation-based structure) and CSV (tabular format) into a cohesive system that LLMs can parse naturally.[1]

users[3]{id,name,role}:
  1,Alice Smith,admin
  2,Bob Jones,user
  3,Carol White,moderator

The syntax breaks down as:

users — Collection name
[3] — Number of items (optional but recommended)
{id,name,role} — Field names declared once
Subsequent lines contain comma-separated values
2-space indentation for nesting

Comparison with JSON

//JSON format (117 tokens):
{
  "items": [    {
      "sku": "A1",
      "name": "Widget",
      "qty": 2,
      "price": 9.99
    },
    {
      "sku": "B2",
      "name": "Gadget",
      "qty": 1,
      "price": 14.5
    },
    {
      "sku": "C3",
      "name": "Doohickey",
      "qty": 5,
      "price": 7.25
    }
  ]
}

// TOON format (49 tokens):
items[3]{sku,name,qty,price}:
  A1,Widget,2,9.99
  B2,Gadget,1,14.5
  C3,Doohickey,5,7.25

This example demonstrates a 58.1% token reduction (68 fewer tokens) for identical information.[1] The efficiency comes from eliminating repeated field names and reducing structural overhead.

How LLMs Understand TOON Without Training

A critical question emerges: if TOON was created in 2025, how can existing LLMs understand it when they weren’t trained on this format?

The answer lies in transfer learning. Even without specific training, LLMs can understand TOON because it borrows familiar patterns from formats they were trained on (specifically YAML and CSV).[1]

“TOON works best when you show the format instead of describing it. The structure is self-documenting — models parse it naturally once they see the pattern. Models treat it like familiar YAML or CSV.”[1]

This understanding capability means TOON can be deployed immediately without fine-tuning.

Using TOON in Practice

Implementation Libraries

TOON has many community driven production-ready implementations across multiple programming languages:

Python: python-toon by xaviviro[7]
Python: toon-llm by davidpirogov[8]

from toon import parse, stringify
# Parse TOON to Python objects
data = parse("""
products[2]{id,name,price}:
  101,Laptop,999.99
  102,Mouse,29.99
""")
# Access data
print(data['products'][0]['name'])  # Output: Laptop
# Convert Python objects to TOON
users = {
    'users': [        {'id': 1, 'name': 'Alice', 'role': 'admin'},
        {'id': 2, 'name': 'Bob', 'role': 'user'}
    ]
}
toon_string = stringify(users)

JavaScript/TypeScript: Official reference implementation[1]

import { parse, stringify } from 'toon-format'
// Parse TOON string
const data = parse(`
orders[3]{orderId,customer,total}:
  ORD-001,John Smith,150.00
  ORD-002,Jane Doe,275.50``
  ORD-003,Bob Wilson,89.99
`)
// Access data
console.log(data.orders[0].customer) // Output: John Smith
// Convert to TOON
const inventory = {
  items: [    { sku: 'A1', stock: 50, location: 'Warehouse-A' },
    { sku: 'B2', stock: 23, location: 'Warehouse-B' }
  ]
}
const toonString = stringify(inventory)

PHP: toon-php by HelgeSverre[9]

use Toon\Parser;
use Toon\Formatter;
// Parse TOON
$parser = new Parser();
$data = $parser->parse("
employees[2]{id,name,department}:
  1001,Alice Johnson,Engineering
  1002,Bob Smith,Marketing
");
// Access data
echo $data['employees'][0]['name']; // Output: Alice Johnson
// Format to TOON
$formatter = new Formatter();
$toonString = $formatter->format([    'tasks' => [        ['id' => 1, 'title' => 'Review PR', 'status' => 'pending'],
        ['id' => 2, 'title' => 'Deploy', 'status' => 'complete']
    ]
]);

Dart: toon by wisamidris77[10]

import 'package:toon_dart/toon_dart.dart';
void main() {
  final data = {
    'user': {
      'id': 123,
      'name': 'Ada',
      'tags': ['reading', 'gaming'],
      'active': true,
      'preferences': []
    }
  };
  print(toonEncode(data));
}

Go: gotoon by alpkeskin[11]

package main
import (
    "fmt"
    "log"
    "github.com/alpkeskin/gotoon"
)
func main() {
    data := map[string]interface{}{
        "users": []map[string]interface{}{
            {"id": 1, "name": "Alice", "role": "admin"},
            {"id": 2, "name": "Bob", "role": "user"},
        },
    }
    encoded, err := gotoon.Encode(data)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(encoded)
}

When to Use TOON

Tabular Data with Uniform Objects
RAG System Contexts [2]
Analytics Dashboards
AI-Powered Classification [2]

When NOT to Use TOON

1. Deep Nesting Required

2. Inconsistent Object Structures

# This is awkward in TOON
mixed_data[3]{id,name,email,phone,address}:
  1,Alice,alice@example.com,,,  # Missing phone and address
  2,Bob,,555-1234,,              # Missing email and address
  3,Carol,carol@example.com,,123 Main St

3. Tool Calling / Function Definitions

TOON is designed as an input format for data, not for defining tool schemas or function signatures.[3] All major LLM providers (OpenAI, Anthropic, Google) expect tool definitions in JSON format following OpenAPI specifications.[4][5]

Use JSON for:

Function calling schemas
API tool definitions
LLM output when integrating with existing systems

Conclusion

TOON represents a targeted optimization for a specific problem: efficiently passing structured, tabular data to LLMs. It doesn’t replace JSON across the board nor does it try to. Instead, it offers a specialized tool that, when applied to appropriate use cases, delivers measurable token reduction and cost savings.

The format’s compatibility by existing LLMs, combined with production-ready implementations across multiple languages, makes it accessible for rapid integration. As the AI ecosystem grows and token efficiency becomes increasingly important, formats like TOON demonstrate significant operational improvements.

For teams operating LLM-powered systems at scale, particularly those working with tabular data, RAG systems, or analytics applications, TOON deserves consideration as a optimization strategy.

References

[1]: Johann Schopplich, “TOON — Token-Oriented Object Notation”, GitHub Repository, https://github.com/johannschopplich/toon

[2]: “TOON Format Guide: Reduce LLM Token Usage by 50%”, Nihar Daily, https://www.nihardaily.com/131-token-oriented-object-notation-toon-your-path-to-50-token-savings

[3]: Abdulkader Safi, “TOON: The Token-Efficient Data Format for LLM Applications”, https://abdulkadersafi.com/blog/toon-the-token-efficient-data-format-for-llm-applications-complete-guide-2025

[4]: “Function Calling with LLMs”, Prompt Engineering Guide, https://www.promptingguide.ai/applications/function_calling

[5]: “Tool/function calling”, LangChain Documentation, https://python.langchain.com/v0.1/docs/modules/model_io/chat/function_calling/

[6]: Joyal Saji, “The Hidden Cost of Tokens in LLMs”, Medium, November 2025, https://medium.com/@joyalsaji/the-hidden-cost-of-tokens-in-llms-and-how-toon-smarter-strategies-can-shrink-it-827160c92787

[7]: Xavi Viró, “python-toon”, GitHub Repository, https://github.com/xaviviro/python-toon

[8]: David Pirogov, “toon-llm”, GitHub Repository, https://github.com/davidpirogov/toon-llm

[9]: Helge Sverre, “toon-php”, GitHub Repository, https://github.com/HelgeSverre/toon-php

[10]: Wisam Idris, “toon”, GitHub Repository, https://github.com/wisamidris77/toon

[11]: Alp Keskin, “gotoon”, GitHub Repository, https://github.com/alpkeskin/gotoon