Most LLM providers charge by the token, a pricing model that directly couples cost to prompt length and generation size. For developers, this means every extra sentence in a system prompt, every retrieved document in a RAG pipeline, and every turn in an agent loop incrementally raises the bill. Understanding how token-based pricing accumulates is essential for forecasting costs and choosing an inference backend that matches your workload geometry.
What Is a Token?
A token is the atomic unit of text processing for modern transformers. Depending on the tokenizer, one token roughly equals three-fourths of a word, a single punctuation mark, or a fragment of a compound word. Providers bill on two axes: input tokens, which include your system prompt, user message, and any attached context, and output tokens, the generated completion. Both axes matter, but input volume is often the silent cost driver because it grows with every document chunk, image caption, or prior conversation turn you attach.
How Token-Based Pricing Works
Under token-based pricing, the final charge is a function of total tokens consumed multiplied by a per-token rate. This model is used by major open-source inference hosts including Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale. The appeal is granularity: you pay strictly for what you consume. The tradeoff is variance. A single API call with a 100,000-token legal brief in the context window can cost substantially more than a call with a one-sentence prompt, even if both return a 50-token answer. As context windows expand to 128,000 tokens or more, the ceiling for a single request rises in direct proportion.
The Hidden Cost of Context Growth
Agentic and long-context workloads amplify token-based costs in ways that are easy to underestimate. In a ReAct-style agent loop, each tool invocation typically resends the entire conversation history, system instructions, and tool schemas. Over ten iterations, the same static prompt text is billed repeatedly.
Here is a minimal Python illustration showing how input tokens compound across turns when the full history is retained:
def estimate_agent_tokens(system_prompt: str, tools_schema: str, turns: int, avg_reply: int):
# Each turn resends system prompt, tools, and all prior history
base = len(system_prompt.split()) + len(tools_schema.split())
total_input = 0
total_output = 0
history = 0
for _ in range(turns):
turn_input = base + history
total_input += turn_input
total_output += avg_reply
history += avg_reply # assistant reply becomes next turn's history
return total_input, total_output
system = "You are a coding assistant. ..." # ~150 words
tools = "{...}" # ~400 words of JSON schema
inp, out = estimate_agent_tokens(system, tools, turns=10, avg_reply=80)
print(f"Estimated input words: {inp}, output words: {out}")
Under token-based billing, that linearly growing input volume produces a superlinear cost curve because you are charged for every repeated word.
A Request-Based Alternative
Oxlo.ai departs from token arithmetic entirely. The platform uses request-based pricing: one flat cost per API call regardless of how many tokens are in the prompt or the reply. For workloads that send long documents, maintain large conversation buffers, or iterate through multi-step agent workflows, this model removes the penalty for context depth.
Oxlo.ai offers 45+ open-source and proprietary models across 7 categories, including long-context specialists such as DeepSeek V4 Flash with a 1,000,000-token context window, Kimi K2.6 with 131,000 tokens of context, and the DeepSeek R1 671B MoE reasoning model. Because the cost is fixed per request, filling a 128,000-token context window costs the same as sending a single sentence. This predictability makes capacity planning straightforward, and popular models are served with no cold starts.
Switching to Oxlo.ai requires no SDK migration. The platform is fully OpenAI SDK compatible. You only need to change the base URL and API key.
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
A 50,000-token RAG prompt costs the same flat request fee
as a 50-token greeting
response = client.chat.completions.create(
model="deepseek-v4-flash",
messages=[
{"role": "system", "content": "You are
Top comments (0)