Myth FAQ: A Prompt That Fits Is Not a Prompt That Was Read

Your request returns 200. Your answer looks confident. Then the field you needed is missing.

You blame the model. You raise the context limit in your config. Nothing changes. Why?

Because "it fit" and "it was read" are two different claims. Let's separate them.

Myth 1: If the API accepted it, every token was read

A 200 means the transport worked. It says nothing about content.

Truncation happens in three places: your client, a gateway, or the provider. Only one of those is usually your code. Instrument all three anyway.

Evidence you can collect in a single run:

1. Character count of the exact string you sent.
2. Token count reported by the provider in the response usage block.
3. Token count from the tokenizer you assume the provider uses.

Log all three. If the third exceeds the second, you were cut somewhere.

A cheap logging wrapper:

import logging

payload = build_context(blocks, window=128_000, output_reserve=2_000)
logging.info(
    "event=request chars=%d est_tokens=%d blocks=%d",
    len(payload), approx_tokens(payload), len(blocks),
)

Search your logs for that event later. Truncation becomes visible instead of mysterious.

Myth 2: The context window is yours for instructions

Many APIs share one window between input and output. That is the whole trick.

Reserve output budget first. Then spend what remains on input.

Ask for a long answer against a full prompt and watch it stop early. That is not a lazy model. That is arithmetic.

Quick sanity check from the shell:

python -c "print(len(open('prompt.txt').read()))"   # characters you are sending

Then read the usage numbers your provider returns. Compare. Do not guess.

Myth 3: More context always means better answers

This is the expensive myth. Longer contexts add distractors, not knowledge.

Liu et al. ("Lost in the Middle", arXiv:2307.03172) found accuracy is highest when the relevant passage sits at the start or the end of the context. Middle-position retrieval degrades, even when the fact is present the whole time.

So a bigger window can make answers worse. That sounds wrong until you measure it.

Ask yourself one question before every padding decision. Does this block change the answer? If not, it is noise plus risk.

Myth 4: Truncation is the only silent failure

Your own assembler drops blocks too. A [:8000] slice never raises an exception.

A retrieval step that returns zero documents also fails silently. Both produce clean requests with missing facts.

Fix the class of bug, not one instance. Make the assembler fail loud and early.

Myth 5: Free model access changes the correctness math

This is where a free tier earns its place, and also where it misleads you.

Free calls remove price anxiety. They do not remove truncation, dilution, or bad retrieval.

Lower per-call cost changes what you can afford to test. That is the entire real benefit.

That is why I ran my context sweeps against MonkeyCode's free model access. The point was never the price tag. The point was running the same prompt at five positions and ten repetitions without a budget meeting first. I also used the free server option to run the sweep as a script, instead of a notebook I babysit.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability claims about the free models and the free server are operator-supplied. I am not claiming specific quota, model names, or permanence here. Verify them before you plan around them.

Artifact: a context assembler that refuses to lie

The core idea: every block carries a priority. If a block cannot fit, you choose the failure mode deliberately.

from dataclasses import dataclass

def approx_tokens(text: str) -> int:
    return max(1, len(text) // 4)  # proxy only; swap in your real tokenizer

@dataclass
class Block:
    name: str
    text: str
    priority: int  # 1 = keep first, 3 = drop first

class BudgetError(Exception):
    pass

def build_context(blocks, window, output_reserve):
    usable = window - output_reserve
    if usable <= 0:
        raise BudgetError("output reserve consumed the window")
    kept, dropped = [], []
    for b in sorted(blocks, key=lambda b: b.priority):
        cost = approx_tokens(b.text)
        if cost <= usable:
            kept.append(b)
            usable -= cost
        else:
            dropped.append((b.name, cost))
    if dropped:
        raise BudgetError(f"blocks did not fit: {dropped}")
    return "\n\n".join(b.text for b in kept)

Two design choices matter more than the code.

1. The proxy tokenizer is deliberately crude. Replace it with the real one.
2. Over-budget means an exception, not a slice.

If you prefer shrinking over failing, log the dropped list at warning level. Never drop in silence.

The needle test: 40 minutes, reproducible

Goal: measure whether context position changes correctness for your prompt.

1. Write one fact only your repo knows, such as a magic constant.
2. Build one prompt. Place the fact at 5%, 50%, and 95% of the context.
3. Pad with real files, not random text. Distractors should look plausible.
4. Run each variant 10 times with identical settings.
5. Record the answer and the provider's reported prompt tokens.

Results table to fill in:

Position	Runs	Correct	Prompt tokens reported	Tokens you expected
5%	10
50%	10
95%	10

How to read it:

• All positions correct, tokens match: your context assembly is fine.
• Middle position fails: budget time for ordering or retrieval work.
• Reported tokens below your estimate: you are being truncated somewhere.

Do not skip step 5. The token columns are the part people omit, and they explain the most failures.

Decision table: what earns a seat in context

Item	Include?	Why
Stack trace trimmed to the failing frame	Yes	Highest signal per token
Full dependency lockfile	No	Huge and rarely causally relevant
The diff under review	Yes	It defines the task
Entire repository	No	Adds distractors and truncation risk
Last failing test output	Yes	Cheap and targeted
Meeting notes	Only if cited	Unverified context, high noise

Rule of thumb: if you cannot say why a block changes the answer, drop it.

Limitations and who should skip this

This method measures behavior. It does not hand you guarantees.

Findings from one model and one configuration do not transfer automatically. Char-to-token ratios vary by language, code density, and tokenizer.

Skip this approach if you need contractual determinism. Skip it if your data cannot leave your environment. Skip it for regulated workloads with unclear retention rules.

And skip it if a cheaper fix exists. Sometimes the real bug is just a bad prompt.

What to do next

Log three numbers on the next request you send: characters, reported tokens, expected tokens. Do that once and you will never fully trust "it fit" again.

If you want somewhere to run these sweeps without per-call cost anxiety, the free model access on MonkeyCode is where I ran mine. Run the needle test first. Decide after you have rows in that table.