You do not need to train models to use AI well, but you do need to understand eight things: what a model actually is (a file full of numbers), how text becomes tokens, why the context window is working memory rather than knowledge, what a system prompt and an AI harness really do, how input, output and cached tokens are priced, why subscriptions usually beat pay-as-you-go APIs for heavy daily use, what reasoning effort costs you, and where guardrails and permission modes sit. This guide explains each one with current, verified prices and the practical habits that follow from them.
TL;DR - Last verified: 2026-08-23
- An LLM is a giant file of numbers (weights). You never change it by chatting; you steer it through the context window.
- Tokens are the billing unit and the memory unit. Output tokens are the expensive kind.
- Cached input tokens are roughly 10x cheaper than fresh input, so long sessions cost less than you fear, but more than zero.
- New task? New session. Leftover context makes every later reply slower, pricier and less accurate.
- Reasoning effort is a dial that burns output tokens. Leave it low or medium unless the problem is genuinely hard.
What is a large language model, really?
A large language model is, at its core, a very large collection of numbers called weights, plus a small amount of inference code that runs them. When people say a model "knows" something, they mean that information was baked into those weights during training. GPT-5.6, Claude Opus 5, Kimi K3 and GLM-5.2 are all, mechanically, the same kind of object: a frozen set of weights deployed on someone else's GPUs.
That has one consequence most developers miss at first: chatting with a model never changes the model. Your conversation does not teach it, retrain it, or personalize the shared weights for anyone else. Everything a model appears to "remember" during a session is happening somewhere else entirely, in the context window.
Because the weights are frozen, the only lever you have at runtime is what you feed in. That is why so much of practical AI skill comes down to managing input rather than "prompting tricks," and why context engineering has become the real production skill.
What is the AI harness between you and the model?
When you use a coding assistant like Claude Code, Codex or Copilot, you are not talking to the model directly. You are talking to an AI harness - convenience software that wraps the model, gives it tools (read files, run commands, call APIs), and precedes your messages with a system prompt: a high-level instruction that tells the model what job it is doing.
A base model can do almost anything; a coding harness narrows it to programming by defaulting it to behave like "a careful engineer working on this repository." Two things follow:
- The harness inflates your context. That is why a fresh coding session can show ~18K tokens used after you have typed only "hello" - the system prompt and tool definitions count against the window.
- The system prompt is a strong instruction, not a law. Models are trained to weight it heavily, but it can be overridden or jailbroken. Treat it as configuration, not security. Real security lives in guardrails and permission modes (below).
Choosing and configuring harnesses is a topic of its own; our Codex vs Claude Code comparison goes deep on the two biggest developer options.
What are tokens, and why does everything revolve around them?
Tokens are the pieces of text a model actually reads and writes. A tokenizer - a fully deterministic program with no intelligence of its own - chops your text into chunks the model can turn into numbers. Sometimes a token is a whole word, sometimes a fragment, sometimes a leading space plus characters. Long numbers often get split into several tokens, which surprises people.
Why you should care:
- Tokens are the billing unit. Every API price is quoted per million tokens.
- Tokens are the memory unit. Context windows are measured in tokens, not words or pages.
- Rough mental math: for English prose, 1 token is about 0.75 words. Code, non-English text and numbers tokenize worse, so budget more.
What is a context window, and why does less context beat more?
The context window is the model's working memory: the maximum number of tokens it can attend to in one request. The model's deep knowledge lives in the weights (fixed at training); the context window is a scratchpad where your conversation, files and tool results live for the duration of the session.
Three practical rules drop out of this:
- Smaller context = sharper model. Every frontier family offers huge windows now, but providers deliberately cap practical coding sessions (you will see limits like ~258K in tools) because attention quality degrades as the window fills. A model with 20K tokens of clutter performs worse than the same model with 5K tokens of focus.
- Old tasks poison new ones. If you finish Task A and start unrelated Task B in the same session, the model re-reads all of Task A's context on every turn. It costs more and answers worse.
- The fix is boring: start a fresh session per task. Keep sessions short, and export anything worth keeping into a file the model can re-read on demand.
We tested versions of this advice across real agent setups in our guide to why you can't trust AI-generated code without good context hygiene.
How does LLM pricing work (input, output, cached)?
Every request bills you in up to four buckets. Using Claude Fable 5 as a worked example (prices per million tokens, verified against Anthropic's pricing page August 2026):
| Bucket | What it is | Fable 5 price |
|---|---|---|
| Input (fresh) | New tokens you send for the first time | $10.00 |
| Cache write (5-min TTL) | Storing a reusable prefix | $12.50 (1.25x input) |
| Cache write (1-hour TTL) | Same, longer-lived | $20.00 (2x input) |
| Cache read / hit | Re-reading a cached prefix | $1.00 (90% off) |
| Output | Tokens the model generates | $50.00 |
The mechanics that matter:
- Conversation history is re-billed every turn - as cached input. When you send message #3, messages #1 and #2 ride along as input. With prompt caching they bill at the discounted read rate (about 10% of fresh input); without it, they bill at the full input rate. OpenAI's GPT-5.6 family works the same way, with cached input at 10% of the standard rate (GPT-5.6 Sol lists at $5.00 input / $30.00 output per million).
- Output tokens are always the most expensive bucket (typically 4-8x input). Anything that makes the model write less - tighter instructions, structured output, lower reasoning effort - moves the bill more than trimming your prompt.
- Caches expire. Prompt caching is ephemeral by design (5-minute or 1-hour TTLs). A cache hit refreshes the TTL; an expired cache silently becomes fresh input and costs 10x more.
Should you pay per token (API) or per month (subscription)?
For heavy daily interactive use, subscriptions win. Frontier labs sell two things: pay-per-token API access, and flat monthly plans (roughly $20 to $200/month depending on tier) that bundle large token quotas behind weekly or 5-hour rate limits. The bundles are priced far below what the same token volume would cost through the API - it is common for a $20/month plan to deliver usage that would be hundreds or thousands of dollars at list API prices.
The mental model:
- Interactive daily work (coding assistant, research, drafting) -> a subscription. Predictable, cheap, rate-limited.
- Programmatic workloads (your app calling the model at 3am, batch jobs, customer-facing features) -> the API. You need programmatic access, and you should combine prompt caching, batch endpoints (typically 50% off) and a hard spend limit from day one.
- Never confuse the two. A consumer subscription does not include API access; they are separate bills.
What do reasoning ("thinking") modes actually do?
Reasoning modes give the model a second scratchpad on top of the context window: before answering, the model generates a chain of intermediate reasoning that you may or may not see. Higher reasoning effort means more of that deliberation per turn - and better answers on genuinely hard problems.
The cost structure is what matters to you:
- Reasoning tokens are output tokens. You pay the top-tier output rate for all that hidden thinking - on Fable 5 that is $50 per million.
- Effort is a dial, not a switch. Current harnesses expose levels (low, medium, high, extra high). Low and medium exist because most tasks are not hard enough to justify burning triple the output budget.
- Default low; escalate on evidence. Turn effort up when you can see the shallow answer failing, not as a blanket policy.
Where do guardrails and permission modes fit?
Safety lives at two layers, and developers routinely conflate them:
- Model-level guardrails are trained in by the lab: the model itself refuses to help with malware, bio-threats, and similar. These travel with the weights no matter what harness you use. They are strong but not absolute - treat them as a filter, not a guarantee.
- Harness-level permission modes govern what the agent may do on your machine: run shell commands, edit files, hit the network. Modern coding agents offer a spectrum from "approve every action" through auto-gated modes (a second model classifies each action) to fully autonomous "bypass" modes, plus read-only plan mode where the agent may only propose, never execute.
Practical policy for real work: plan mode for exploration, an auto-gated mode for daily driving, manual approval for anything touching production or credentials. Bypass permissions exist for sandboxes and throwaway containers - nowhere else.
What this means for you
- Start a new session per task. It is the single cheapest quality upgrade available.
- Write less, fetch precisely. Keep prompts tight; make the harness pull only the files that matter; keep context below ~30% of the window when you can.
- Cache the stable parts. Put your big system prompt and reference docs at the front so prompt caching can discount them by ~90%.
- Leave reasoning effort on low/medium unless you are staring at a genuinely hard problem.
- Put a hard spend limit on every API key before your first production request. Retry loops are how $50 estimates become $5,000 invoices.
- Never run an agent in bypass mode on a machine with real credentials or un-backed-up work.
FAQ
Q: What is a token in an LLM, in plain terms?
A: A token is a chunk of text - a word, word fragment, or space-plus-characters - that the model can turn into a number. English prose averages about 0.75 words per token. Tokens are both the billing unit (prices are per million tokens) and the memory unit (context windows are measured in tokens).
Q: Does chatting with an AI model train or improve it?
A: No. Production chat sessions never modify the model's weights. What looks like "learning" during a conversation is just the context window - everything in it is forgotten when the session ends unless the product explicitly stores memory elsewhere.
Q: Why are output tokens so much more expensive than input tokens?
A: Reading input is a parallel operation over your prompt, while generating output is sequential - each token requires a full forward pass through the model. Output typically costs 4-8x input (for example Claude Fable 5 is $10 per million input vs $50 per million output), and hidden reasoning tokens bill as output too.
Q: How much does prompt caching actually save?
A: Cache reads bill at roughly 10% of the standard input rate from both Anthropic and OpenAI. For workloads that resend a large stable prompt every turn - chatbots, agents, coding sessions - that is about a 90% discount on the repeated portion of the input.
Q: Is an AI subscription cheaper than using the API?
A: For heavy daily interactive use, yes by a wide margin; monthly plans bundle token quotas that would cost many times more at API list prices. For automated, programmatic workloads you need the API regardless - subscriptions cannot be used that way.
Q: What is the difference between a guardrail and a permission mode?
A: A guardrail is trained into the model and decides what kinds of content it will produce (refusing malware requests, for example). A permission mode is enforced by the harness and decides what actions the agent may take on your machine - run commands, edit files, access the network. You need both; neither substitutes for the other.
Sources
- Anthropic API pricing - per-million-token rates and cache pricing for Claude models (verified 2026-08-23).
- OpenAI API pricing - GPT-5 family rates and cached-input discounts (verified 2026-08-23).
- Anthropic prompt caching documentation - cache TTLs, write costs and hit pricing mechanics.
Updates log
- 2026-08-23 - Initial publication. Prices verified against official Anthropic and OpenAI pricing pages.
Top comments (0)