Most LLM APIs bill by the token. If you are building agents, processing long documents, or running multi-turn conversations, that unit of measurement becomes the single most important variable in your monthly cloud bill. This guide explains how token-based pricing is structured, where hidden costs appear, and how to model your spend before shipping to production. We will also look at request-based alternatives, including Oxlo.ai, that remove token counting entirely.
What Is a Token?
A token is not a word. It is a sub-word unit produced by a byte-pair encoding (BPE) or similar tokenizer. A short English word like "hello" might be one token, while a rare word like "tokenization" can split into three or four pieces. Punctuation, whitespace, and code symbols each consume tokens, and multilingual text often expands into more tokens per character than English.
You can verify this locally with OpenAI's open-source tiktoken library:
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
text = "Token-based pricing charges per sub-word unit."
tokens = enc.encode(text)
print(f"Token count: {len(tokens)}")
print(f"Token IDs: {tokens}")
Running this gives a token count of 9 for 49 characters. Because billing is tied to these integer IDs, two prompts of identical character length can yield different charges depending on vocabulary frequency and language.
How Token-Based Pricing Works
Token-based providers, including Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, typically split pricing into two streams:
- Input tokens: Every token sent to the model, including system instructions, conversation history, file uploads, and few-shot examples.
- Output tokens: Every token generated by the model and streamed back to your client.
Both streams are metered separately. Most providers publish a rate per million tokens for each direction. When you send a request, the platform counts input tokens before inference begins, then counts output tokens as they are generated. Your bill is the sum of the two.
A subtle but expensive detail is that the entire context window is re-billed on every turn. In a 20-turn conversation, you do not pay for the latest message alone. You pay for the system prompt plus all 20 turns of history every single time. For agents that append tool results and chain-of-thought reasoning into the context, input token volume grows quadratically.
Estimating Real-World Costs
The standard formula is straightforward:
def estimate_cost(input_tokens, output_tokens, input_rate, output_rate):
"""
Rates are denominated in cost per 1,000,000 tokens.
"""
return (input_tokens * input_rate + output_tokens * output_rate) / 1_000_000
Where this becomes painful is in the sensitivity to input length. A retrieval-augmented generation (RAG) pipeline that injects five chunks of text can easily reach 8,000 to 16,000 input tokens. Add a detailed system prompt and multi-turn history, and a single agentic step can push past 32,000 input tokens. Because input rates apply to every token in the context window, long-context workloads produce disproportionate bills before the model emits a single completion token.
Where Token Bills Grow
Beyond simple prompt and completion counts, four patterns silently inflate token spend:
- Repeated system prompts: A 1,000-token system message is billed on every request in a session, not once.
- Conversation history: Multi-turn chat requires resending the full message array. A 10-turn dialogue with 500 tokens per turn becomes a 5,000-token input on the final call.
- Agentic loops:
Top comments (0)