Most AI inference providers bill by the token. It is the industry standard, but it is also a source of unpredictable costs, especially when prompts grow long or agents loop repeatedly. If you are building applications with large context windows or multi-step tool use, understanding exactly how token-based pricing works is the first step toward controlling your bill. For many workloads, an alternative model, request-based pricing, eliminates the guesswork entirely.
What Is a Token?
A token is the atomic unit of text that a language model processes. It is not a word. Depending on the tokenizer, a single word might be one token, multiple tokens, or a fraction of one. Punctuation, whitespace, and Unicode characters all consume tokens. As a rough rule, one token corresponds to about 0.75 words in English text, but this varies by model family.
Code is typically less efficient. A single line of Python with indentation and symbols can consume significantly more tokens than a line of prose. Image inputs, when supported, are often converted into token-equivalent patches or sequences that are counted against your context limit and your budget.
# A simple heuristic for estimating token count
# Note: this is an approximation. Always use the provider's tokenizer for accuracy.
def estimate_tokens(text):
return int(len(text.split()) * 1.33)
prompt = "System: You are a helpful assistant.\\nUser: Summarize the attached legal brief."
print(f"Estimated tokens: {estimate_tokens(prompt)}")
How Token-Based Pricing Works
Token-based pricing is split into two buckets: input tokens and output tokens. Input tokens include everything you send to the model: system instructions, conversation history, tool definitions, retrieved documents, and the current user message. Output tokens are what the model generates in response.
Providers typically charge different rates for each. Input tokens are often cheaper per unit than output tokens, but because you pay for the entire context window on every request, costs scale linearly with prompt length. If you send a 100,000-token document and ask for a one-sentence answer, you pay for 100,000 input tokens plus whatever the model generates.
Some providers also distinguish between cache hits and misses, or offer discounts for prompt prefix caching. These optimizations can reduce costs, but they add complexity to your billing logic and are not universally available.
The Hidden Costs of Long Context and Agents
The real challenge with token-based billing is not the single request. It is the cumulative weight of context over time. Agentic applications that maintain multi-turn state, call tools, and append results back into the prompt cause context length to balloon. Every intermediate reasoning step, every JSON schema, and every retrieved chunk adds to the input token count.
For example, a coding agent that iterates five times over a repository with 50,000 tokens of context will pay for that 50,000-token context five times over, plus any generated reasoning tokens on each turn. The same dynamic applies to RAG pipelines that stuff multiple retrieved passages into the prompt, or to conversational UIs that keep full history.
Because output length is stochastic, forecasting monthly spend requires modeling token distributions. A burst of verbose outputs or a logic error that triggers a retry loop can spike costs unpredictably.
Cost Estimation and Budgeting
Engineers usually estimate costs with a simple formula:
total_cost = (input_tokens * input_rate) + (output_tokens * output_rate)
In practice, this equation is fragile. Input tokens change every time you modify a system prompt or add a tool. Output tokens vary by temperature, top-p settings, and the complexity of the task. If your application processes variable-length documents, you need to build percentile models of document size to avoid surprises.
Token-based providers often publish rate cards, but those rates are subject to change, and they differ across model tiers. A developer switching from a 70B model to a 400B+ mixture-of-experts model might see the per-token rate double or triple alongside the context window expansion. Budgeting becomes a forecasting exercise rather than a hard cap.
When Token-Based Pricing Fits
Token-based billing is not without merit. For short, stateless requests, it can be economical. If your workload consists of classification tasks, single-turn Q&A, or structured extraction from small inputs, token counts are low and predictable. In these cases, paying per token aligns cost closely with compute consumed.
It also incentivizes prompt efficiency. Teams that optimize their prompts, prune conversation history, and compress retrieved context directly reduce their spend. For applications with tight engineering resources and simple request patterns, token-based pricing is a straightforward, widely supported option.
When Request-Based Pricing Wins
For long-context workloads and agentic systems, the linear scaling of token costs becomes a liability. This is where request-based pricing changes the equation. Instead of metering input and output tokens, you pay one flat cost per API request regardless of prompt length.
Consider a workflow that passes a 200,000-token codebase to a model for architectural review. Under token-based billing, the input cost alone is substantial, even if the model returns a brief analysis. Under request-based billing, the cost is identical whether the prompt is 100 tokens or 100,000 tokens. The same logic applies to multi-turn agents that accumulate tool results and conversation state. The cost per step remains constant, making budgets deterministic.
Oxlo.ai is built around this model. It offers flat per-request pricing that can be 10-100x cheaper than token-based alternatives for long-context and agentic workloads. Because cost does not scale with input length, you can pass full documents, maintain long conversation histories, and run complex tool chains without watching a meter spin.
Oxlo.ai's Developer-First Platform
Oxlo.ai provides 45+ open-source and proprietary models across seven categories, all accessible through a fully OpenAI SDK compatible API. There are no cold starts on popular models, and the base URL is a simple drop-in replacement.
import openai
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
# One flat request cost, regardless of how long the prompt is
response = client.chat.completions.create(
model="deepseek-r1-671b",
messages=[
{"role": "system", "content": "You are a senior software architect."},
{"role": "user", "content": long_codebase_context} # 100k+ tokens
]
)
The platform includes flagship models such as DeepSeek R1 671B MoE for deep reasoning, Kimi K2.6 for agentic coding and vision, Llama 3.3 70B for general-purpose tasks, and DeepSeek V4 Flash with a 1M context window. Categories span LLMs, code models, vision, image generation, audio, embeddings, and object detection. Features like streaming, function calling, JSON mode, and vision input are all supported.
Pricing is tiered by requests per day, not tokens. The Free plan includes 60 requests per day and access to 16+ free models, including DeepSeek V3.2. Paid plans scale to thousands of requests per day with priority queue access at the Premium level. Enterprise customers receive custom unlimited deployments with dedicated GPUs. For exact plan details, see the Oxlo.ai pricing page.
Optimizing Token Usage
Even if you use a token-based provider, or if you simply want to reduce latency on Oxlo.ai, token discipline matters. Shorter prompts parse faster and reduce time-to-first-token. Here are practical techniques:
- Truncate conversation history: Keep only the most recent turns, or summarize older turns into a compressed system message.
- Structure retrieved context: In RAG systems, rerank and deduplicate chunks before stuffing them into the prompt. Quality beats quantity.
- Use concise schemas: When forcing JSON mode or tool definitions, strip unnecessary whitespace and descriptions from the schema if the model allows it.
- Batch where possible: If a provider supports batching, grouping multiple small tasks into one request can amortize overhead, though this increases per-request latency.
- Compress repetitive prefixes: Some providers cache system prompts or repeated prefixes. If yours does, structure your requests to maximize cache hit rates.
Conclusion
Token-based pricing is the default for a reason. It maps cost to compute, and for small, simple requests, it works well. But it also introduces unpredictability that compounds as applications grow more sophisticated. Long inputs, agentic loops, and large context windows turn token counting into a budget risk.
Understanding how tokens are counted, where costs accumulate, and how to optimize prompts is essential for any team shipping LLM features. For workloads where context length is variable or inherently large, request-based pricing offers a simpler, more predictable alternative. Oxlo.ai delivers that model across a broad catalog of open-source and proprietary models, with full OpenAI SDK compatibility and no cold starts. Evaluate your workload shape, model your token distributions, and choose the pricing structure that aligns cost with value.
Top comments (0)