Meet TOON (Token-Oriented Object Notation): Save yourself money & improve LLM performance
Tired of seeing your Large Language Model (LLM) API costs skyrocket, especially when feeding structured data like user lists or analytics rows? The culprit is often the excessive token consumption from standard formats like JSON. Every brace, bracket, and repeated key eats into your context window and your budget.
TOON, short for Token-Oriented Object Notation, is the newer data serialization format that has been designed specifically to reduce the number of tokens and subsequently the cost associated with LLMs while passing structured data to Large Language Models. TOON format in a way replaces JSON data format to pass structured data to the LLMs. It borrows from CSV and YAML formats to reduce repetitions and indentations are used for better readability
Reasons for Efficiency
Minimal Syntax:
While creating JSON objects a lot of characters are used which increases the numberof tokens utilized by the LLMs, thereby increasing their costs. TOON on the other hand, rather than using characters like {}, use indentation instead, in place of commas, a new line is used, and quotes are avoided unless they are in the strings cotaining special characters.
Example:
JSON:{"name": "Alice", "age": 30}
TOON:
Alice\nage: 30
Tabular Arrays:
TOON uses Tabular arrays to avoid using repeated field names just the way CSV does, to further decrease the token count.
Example:
JSON:[{"id": 1, "name": "Alice"}, {"id": 2, "name":
"Bob"}]
TOON:[2]{id,name}:\n1,Alice\n2,Bob
LLM Specific Guardrails:
TOON uses features like Explicit Length markers (example: [2]) which serves as a hint to the LLM for validtaion and it also explicitly provides the field names upfront to the LLM thereby giving the models a clear schema.
Use Cases
- Input Format: This format is used primarily to send input data to the LLMs
- Structured Data: Whether that be lists of records, log entries or any other kind of structured data has to be sent to LLM
- Efffective utilization of context windows: Since it decreases the token count, it helps in sending the maximum amount of structured data that you can send across in the single context window of an LLM.
You can read more about TOON data format from their Github page linked here