A Bigger Context Window Is Not More Memory: Five Token Myths and a Budget Gate

Your counter says 12k tokens. The API returns a 400. Which one is lying?

Probably neither. They are counting different things. One of them is counting your string. The other is counting what the model actually receives.

I keep seeing the same five assumptions in code review. Each one feels safe. Each one fails quietly, or fails loudly at 2 a.m.

Here is the FAQ, the evidence you can collect yourself, and a small gate you can run in CI.

Myth 1: "One token is about four characters"

This is a rule of thumb for English prose. Vendors describe it that way. It is not a unit conversion.

The ratio moves with content. Code, JSON, CJK text, emoji, and long identifiers all drift. Whitespace and punctuation get their own pieces. So does anything the tokenizer has never seen.

Corrected model: characters are a hint. Tokens are what the model bills and truncates.

You do not need to trust my ratio. Measure yours. Grab 20 real prompts from your logs. Count them with the tokenizer your provider actually ships. Then compare against len(text) / 4.

Different tokenizers disagree with each other too. tiktoken encodings are one family (see the tiktoken repo). Other model families use other vocabularies. A prompt that fits under one tokenizer can overflow under another.

So: which tokenizer does your budget check use?

Myth 2: "The context window is one number"

It is at least three numbers wearing a trench coat.

• Input budget. What the prompt may consume.
• Output budget. What the answer may consume. Often a separate parameter with its own ceiling.
• Template overhead. Role markers, separators, and tool schemas. Not free. Not zero.

Input plus output usually share one window. Reserve the answer before you count the prompt. Most overflow bugs I read are really "nobody subtracted max_tokens".

Tool schemas are the sneaky part. A generous function signature block can cost more than your whole user message. It is sent on every turn.

Corrected model: budget = window − reserved output − template overhead. Then subtract tool schema cost, every turn.

Myth 3: "More context means better recall"

Long-context evaluations keep reporting uneven recall by position. The well-known result is "Lost in the Middle". Details placed mid-prompt get retrieved less reliably than details at the edges.

That is not a reason to panic. It is a reason to stop stuffing.

Corrected model: context is capacity, not attention. A 200k window does not mean 200k of useful signal. Placement and redundancy matter more than volume.

Practical translation: put the constraint you cannot afford to lose near the top. Put the task near the bottom. Do not bury the schema between two chatty tool results.

Myth 4: "Dropping old messages is always safe"

Old messages carry two things. Conversation, and contract.

When you slice messages[2:], you may delete the system constraints. You may also orphan a tool call. Many APIs reject an assistant tool_calls block that has no matching tool result, or the reverse.

That is the 400 you cannot reproduce locally. Your unit test never had the tool pair in the middle.

Corrected model: truncation is a structural edit, not a string edit. Validate pairing after every cut.

Myth 5: "If the API accepted it, the model saw all of it"

Sometimes something upstream trims. A proxy, a gateway, a retry wrapper, an SDK default. The request succeeds. The answer misses the paragraph you cared about.

You cannot detect that from the response alone. You can detect it from the request you sent.

Corrected model: log the estimate at send time. If the answer looks blind, diff the budget against the prompt you thought you sent.

The gate: one file, standard library only

Here is the artifact. It estimates prompt cost, reserves answer space, and checks tool-call pairing. It exits non-zero when the prompt does not fit.

Treat the built-in counter as a conservative estimate. Swap in a real tokenizer when you have one.

#!/usr/bin/env python3
"""context_gate.py - fail fast when a prompt no longer fits the budget."""
import json
import re
import sys
from dataclasses import dataclass


def estimate_tokens(text: str) -> int:
    # Deliberately conservative placeholder. Never exact. Swap in a real tokenizer.
    words = len(re.findall(r"\w+|[^\w\s]", text))
    chars = (len(text) + 3) // 4
    return max(words, chars)


@dataclass
class Budget:
    context: int       # total window, from the provider docs
    reserve: int       # space kept for the answer
    overhead: int = 8  # role markers / separators per message


def count_prompt(messages, budget: Budget, counter=estimate_tokens) -> int:
    total = 0
    for m in messages:
        total += counter(str(m.get("content") or "")) + budget.overhead
        for call in m.get("tool_calls") or []:
            total += counter(json.dumps(call.get("function", {}))) + budget.overhead
    return total


def unpaired_tool_calls(messages):
    pending, problems = [], []
    for i, m in enumerate(messages):
        for call in m.get("tool_calls") or []:
            pending.append(call.get("id"))
        if m.get("role") == "tool":
            cid = m.get("tool_call_id")
            if cid in pending:
                pending.remove(cid)
            else:
                problems.append(f"message {i}: result for unknown call {cid}")
    problems += [f"call {c}: no result" for c in pending]
    return problems


def check(messages, budget: Budget, counter=estimate_tokens):
    used = count_prompt(messages, budget, counter)
    allowed = budget.context - budget.reserve
    return {
        "prompt_tokens_est": used,
        "prompt_budget": allowed,
        "answer_reserve": budget.reserve,
        "fits": used <= allowed,
        "pairing_problems": unpaired_tool_calls(messages),
    }


if __name__ == "__main__":
    path, context, reserve = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])
    with open(path) as f:
        payload = json.load(f)
    report = check(payload["messages"], Budget(context=context, reserve=reserve))
    print(json.dumps(report, indent=2))
    sys.exit(0 if report["fits"] and not report["pairing_problems"] else 1)

Run it against a captured request:

python3 context_gate.py prompt.json 32768 1024

A failing run looks like this:

{
  "prompt_tokens_est": 31895,
  "prompt_budget": 31744,
  "answer_reserve": 1024,
  "fits": false,
  "pairing_problems": []
}

The 32768 is a flag you supply. Read the real number from your provider's docs. Do not copy mine.

Run it before the request, not after

1. Capture one real request body as JSON. Keep it as a fixture.
2. Add your two hardest cases: a large tool schema, and a sliced history.
3. Run the gate on both. Fix the budget, not the fixture.
4. Wire it into CI as a preflight step.
5. Log prompt_tokens_est next to every response you store.

Step 5 is the one people skip. Then the incident review has no numbers.

I run this preflight on MonkeyCode's free server option so the check travels with the job instead of living on my laptop. I also point it at free model access to compare estimates across more than one tokenizer family. That comparison is the whole point — one counter proves nothing.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Cut-first decision table

When the gate says no, do not just slice. Pick the cheapest effective edit.

Symptom	First cut	Why
Tool schema dominates	Prune optional params, shorten descriptions	Paid every turn, not once
Long history, short task	Summarize turns older than N	Keeps the contract intact
One huge document	Move to retrieval, pass top-k	Placement beats volume
Duplicated instructions	Keep one canonical system block	Redundancy is not safety
Answer keeps getting cut	Raise reserve before trimming prompt	Reserve is the real ceiling

Limitations and who should skip this

• The built-in counter is an estimate. It will not match provider billing. Use the provider's tokenizer when precision matters.
• Windows, reserves, and limits change. Read current docs. This gate reads numbers you pass in, never assumptions.
• For multi-megabyte corpora, this is the wrong tool. Use retrieval instead.
• If your prompts are short and single-turn, a gate is overhead. Skip it.
• This checks shape and budget, not answer quality. A prompt can fit and still be wrong.

The takeaway

Context is a budget with a shape. Tokens vary by tokenizer. Output competes with input. Placement beats stuffing. Truncation is structural. And anything upstream can trim silently.

So the next time a 400 lands on your desk, what will you show the reviewer: a guess about characters, or a gate that printed the number?