Our system processes billions of tokens each month from the database alone, we can't use YAML or JSON because they are too token heavy. For flat results (like a database query) we simply return it as CSV (which is even 30% fewer tokens than TOON). We haven't adopted TOON yet (we have our own for structured objects) but its definitely more token friendly than YAML. At the scale we operate, YAML is expensive.
I based my statement on the examples I have seen at that moment. And I agree if you use YAML for tabular data it is expensive. That is why in my other comment I mentioned a switch based on the shape of the data. CSV for tabular data and YAML for hierarchical data.
You can even have CSV in YAML.
people:|id,name,age1,Alice,302,Bob,25
If that wasn't possible, I would go all in for TOON for those mixed cases.
TOON is YAML with hierarchical data. So it doesn't reduces tokens. And as you mention, in the case of tabular data CSV is better.
If you can show where TOON is saving tokens over the smart use of YAML and CSV, I'm glad to stand corrected.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Our system processes billions of tokens each month from the database alone, we can't use YAML or JSON because they are too token heavy. For flat results (like a database query) we simply return it as CSV (which is even 30% fewer tokens than TOON). We haven't adopted TOON yet (we have our own for structured objects) but its definitely more token friendly than YAML. At the scale we operate, YAML is expensive.
I based my statement on the examples I have seen at that moment. And I agree if you use YAML for tabular data it is expensive. That is why in my other comment I mentioned a switch based on the shape of the data. CSV for tabular data and YAML for hierarchical data.
You can even have CSV in YAML.
If that wasn't possible, I would go all in for TOON for those mixed cases.
TOON is YAML with hierarchical data. So it doesn't reduces tokens. And as you mention, in the case of tabular data CSV is better.
If you can show where TOON is saving tokens over the smart use of YAML and CSV, I'm glad to stand corrected.