Most LLM API providers bill by the token. You pay for what you send and what you receive, measured in text chunks that average roughly three quarters of a word. This model appears fair for short queries, but its cost structure punishes long contexts, repeated system prompts, and agentic loops. If your application passes large documents or maintains multi-turn conversation state, the bill scales with every character you include.
How Token-Based Pricing Works
Under a token-based model, providers charge separately for input tokens and output tokens. Input includes your system prompt, user messages, retrieved documents, conversation history, and any structured data you inject into the context window. Output includes the generated response. Providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale operate on this model. Because the entire prompt is reprocessed on every request, any increase in input length directly increases cost.
Hidden Costs of Long Context
The token-based approach creates a tension between richness and economy. A developer who wants to give a model more context, perhaps by including a full codebase or a lengthy retrieval-augmented generation prompt, must pay for that context on every single call. Multi-turn conversations are particularly expensive because most implementations resend the full message history each turn. Agentic workflows compound the problem further: tool outputs, intermediate reasoning traces, and looping logic all inflate the input token count.
Estimating Tokens in Practice
Tokenization is not transparent to the user. A single API call might consume far more tokens than a naive word count suggests. Special characters, formatting, and code blocks expand quickly. The following Python example uses the tiktoken library to demonstrate how a long system prompt and a modest document can inflate the input size before the model generates a single output token.
import tiktoken
def estimate_tokens(text, model_encoding="cl100k_base"):
encoder = tiktoken.get_encoding(model_encoding)
return len(encoder.encode(text))
system_prompt = "You are an expert financial analyst. " * 100
document = open("earnings_report.txt").read() # e.g., 40k tokens
user_question = "Summarize the key risks in Q3."
full_prompt = system_prompt + document + user_question
input_tokens = estimate_tokens(full_prompt)
output_tokens = 800 # estimated response length
total_tokens = input_tokens + output_tokens
print(f"Input tokens: {input_tokens}")
print(f"Total estimated tokens: {total_tokens}")
In a token-based system, this single request incurs charges proportional to that total. If the same document is queried ten times, or if the conversation continues for five turns with full history, the cost multiplies even though the underlying knowledge has not changed.
Where Token Pricing Breaks Down
Token-based billing is manageable for short, stateless prompts, but it becomes unpredictable for production workloads. Consider these patterns:
- Agentic loops: An agent that iterates with tool calls resends its scratchpad, prior outputs, and tool results each cycle.
- Code review: Passing an entire repository or large file context to a coding model like Qwen 3 Coder or DeepSeek Coder requires thousands of input tokens per request.
- Batch processing: Running the same long prompt template across thousands of rows turns minor context bloat into a major budget line item.
In each case, cost variability comes from input length, not business value.
Per-Request Pricing with Oxlo.ai
Oxlo.ai uses a request-based pricing model: one flat cost per API request regardless of prompt length. Unlike token-based providers, cost does not scale with input length, so Oxlo.ai is significantly cheaper for long-context and agentic workloads. You can send a full system prompt, a lengthy document, and a multi-turn history without watching the meter rise on every additional token.
Oxlo.ai is a developer-first platform with 45+ open-source and proprietary models across seven categories, including LLMs and reasoning models such as DeepSeek R1 671B MoE, Kimi K2.6, Llama 3.3 70B, and GLM 5. It is fully OpenAI SDK compatible and requires no code changes beyond pointing the base URL to https://api.oxlo.ai/v1. There are no cold starts on popular models, and features like streaming, function calling, JSON mode, and vision input are all supported.
For developers who want to experiment, the Oxlo.ai free tier includes 60 requests per day across 16+ models, plus a 7-day full-access trial. Paid plans scale from Pro to Enterprise with dedicated GPU options. Exact request pricing is available at https://oxlo.ai/pricing.
Choosing Between Models
Neither pricing model is universally superior. Token-based billing can make sense for applications with extremely short, sparse prompts where output volume dominates. Per-request pricing wins when your workloads exhibit any of the following traits:
- Long or variable input context
- Multi-turn state that is resent each request
- Agentic or tool-using loops
- Batch jobs with heavy prompt templates
If your costs are currently dominated by input tokens, a request-based provider removes the penalty for giving the model more information. Oxlo.ai offers a direct migration path because its API surface is a drop-in replacement for the OpenAI SDK.
Conclusion
Token-based LLM pricing is the industry default, but it is not the only option. Its linear relationship between context size and cost creates a tax on complexity. For long-context inference, agentic systems, and any workload where input size is unpredictable, per-request pricing provides cost stability. Oxlo.ai delivers that model on a fully compatible API with a broad catalog of open-source and proprietary models. Evaluate your token profile, then compare against the flat-rate structure at https://oxlo.ai/pricing.
Top comments (0)