The Context Window Is a Budget, Not a Memory: An FAQ With a Truncation Ledger

Ask three developers how much context their agent has.

You get three numbers. All three are wrong in different ways. The window size is a cap. It is not a memory budget, and it is not what you sent.

Here is the FAQ I keep answering. Each answer ends with a corrected mental model you can test today.

FAQ 1: "The window size is the memory size, right?"

No. The window is a hard ceiling shared by input and output.

Reserve output tokens. Subtract tool schemas. Subtract the system prompt. What is left is your prompt budget.

• Window: the ceiling.
• Reserved output: taken off the top.
• Tool schemas: charged on every single turn.
• System prompt: also charged on every turn.

Corrected model: budget = window - reserved_output - schemas - system.

Do the subtraction before you design the agent. Not after.

FAQ 2: "If the call succeeded, nothing was dropped."

Succeeded means the request was valid. It does not mean it was complete.

Truncation can happen in three places:

1. Your client cuts history before sending.
2. Your framework trims tool results silently.
3. The provider applies its own policy server-side.

You only control the first two. You can still measure all three from the outside.

Corrected model: a 200 response is a transport fact, not a content receipt.

FAQ 3: "Truncation always removes the oldest message."

Sometimes. Not reliably.

Common policies include: drop oldest, drop middle, drop large tool outputs first, or keep system plus the newest N tokens.

The dangerous case is the middle drop. Your first decision disappears. Your latest error survives.

Then the agent re-asks a question you already answered. Nobody notices, because nothing errored.

Corrected model: know your policy by name, or you do not know your policy.

FAQ 4: "Summarize the history and you keep what matters."

Summaries drift. They also lose exact strings.

Identifiers, error codes, column names, and constraints must stay verbatim. Prose summaries mangle them.

Use anchors instead:

• Pin decisions as decision: ... lines.
• Pin identifiers as id=....
• Pin failures as ERROR: ....
• Summarize everything else.

Corrected model: summarize the narration. Never summarize the invariants.

FAQ 5: "A bigger window beats retrieval."

Not for cost, and rarely for accuracy.

Pasting a 40-page doc costs tokens on every turn. Retrieval costs them on one turn.

Small window plus IDs plus a fetch tool is usually the cheaper design. It is also easier to debug.

Corrected model: treat the window as cache, not storage. Storage lives outside the prompt.

The artifact: a truncation ledger

This script counts each turn, simulates a tail-keeping policy, and reports which anchors were lost.

Token counts are a heuristic. They are not a provider measurement. That is deliberate, so it runs anywhere.

# context_ledger.py — count what you send, then simulate a truncation policy.
# Token counts here are a heuristic (~4 chars/token), not a provider measurement.
import re, sys

ANCHOR = re.compile(r'(decision:|id=|ERROR|CONSTRAINT)')

TURNS = [
    {'name': 't1_cold_start', 'messages': [
        {'role': 'system', 'content': 'You are a release assistant. CONSTRAINT: never edit main.'},
        {'role': 'user', 'content': 'decision: use staging-db for the migration dry run'},
    ]},
    {'name': 't2_after_tool_call', 'messages': [
        {'role': 'system', 'content': 'You are a release assistant. CONSTRAINT: never edit main.'},
        {'role': 'user', 'content': 'decision: use staging-db for the migration dry run'},
        {'role': 'assistant', 'content': 'Running the dry run now.'},
        {'role': 'user', 'content': 'ERROR: migration 0042 failed on id=staging-db'},
    ]},
]

def toks(text):
    return max(1, (len(text) + 3) // 4)

def cost(msg):
    return toks(msg['content']) + 4  # role and framing overhead

def truncate_tail(messages, budget):
    system = [m for m in messages if m['role'] == 'system']
    rest = [m for m in messages if m['role'] != 'system']
    used = sum(cost(m) for m in system)
    kept = []
    for m in reversed(rest):
        if used + cost(m) > budget:
            break
        kept.append(m)
        used += cost(m)
    return system + list(reversed(kept)), used

def anchors(messages):
    return {m['content'] for m in messages if ANCHOR.search(m['content'])}

def main(budget):
    print(f'prompt budget = {budget} heuristic tokens')
    for turn in TURNS:
        kept, used = truncate_tail(turn['messages'], budget)
        lost = anchors(turn['messages']) - anchors(kept)
        print('{:<18} used={:<4} headroom={:<4} lost={}'.format(
            turn['name'], used, budget - used, len(lost)))
        for line in sorted(lost):
            print('    LOST: ' + line[:60])

if __name__ == '__main__':
    main(int(sys.argv[1]) if len(sys.argv) > 1 else 200)

Run it twice. Once with a generous budget. Once with a tight one.

python3 context_ledger.py 200
python3 context_ledger.py 40

Illustrative output on the built-in sample:

prompt budget = 200 heuristic tokens
t1_cold_start      used=36   headroom=164  lost=0
t2_after_tool_call used=62   headroom=138  lost=0

prompt budget = 40 heuristic tokens
t1_cold_start      used=36   headroom=4    lost=0
t2_after_tool_call used=35   headroom=5    lost=1
    LOST: decision: use staging-db for the migration dry run

Read the second table again. The agent kept the newest error. It dropped the constraint that made the error meaningful.

That is the bug you will ship. Not a crash. A wrong assumption, confidently stated.

How to use it on your own prompts

1. Replace TURNS with real logged turns, minus secrets.
2. Add anchors for your domain: ticket IDs, currency codes, tenant slugs.
3. Sweep budgets until anchors start disappearing.
4. Record that number. Call it your real budget.
5. Wire an assertion so the pipeline fails when anchors drop.

Step five is the whole point. A ledger you never assert on is a diary.

Choosing a policy

Situation	Policy	What it costs you
Short sessions, small prompts	send everything	linear cost growth
Long chat, stable early decisions	pin anchors, summarize narration	summary drift on details
Large documents	retrieve by ID	occasional recall miss
Tool-heavy loops	drop old tool output first	you lose the earlier error

There is no free row in that table. Pick the failure you can detect.

Running the replay cheaply

A script this small does not need a big machine. It needs a place to run and a model to replay against.

MonkeyCode provides free model access and a free server option (as supplied to me by the operator). That maps well to replay work: run the same turn file after every prompt change, and diff the ledger.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not claiming quotas, hardware specs, or permanence. Check the current terms before you build a pipeline on any free tier.

The useful part is the loop: change a prompt, replay, diff the anchor count. Free or paid, that loop is what catches regressions.

Limitations and who should skip this

Be honest about what the ledger cannot do.

• The token count is a heuristic. Real tokenizers differ, often by 15-20%.
• It simulates your policy, not the provider's hidden one.
• It cannot measure attention quality. A model can receive a line and still ignore it.
• The anchor regex is a stand-in. Real invariants need real schema.
• It says nothing about latency or cost per token.

Skip this approach if you need verbatim transcript retention for audit. Keep the full transcript in external storage and pass IDs into the prompt instead.

Skip it too if you need guaranteed capacity. A free option is a starting point, not a contract.

The question worth asking

Which anchor would your agent lose first?

If you cannot answer that, you do not know your agent. You know its window size.

Run the ledger before the next refactor. Then commit the turn file next to the prompt, so the next change is comparable.

Short scripts. Real budgets. Fewer confident wrong answers.