Last week I pasted three lecture transcripts into a free chat model and asked, "What does Theorem 3 in section five say?" The model answered that section five did not contain a theorem.
The theorem was there, in the second paragraph, about forty thousand tokens into the prompt.
The model was not lying. It had simply stopped attending to that region.
That made me curious: where exactly does a long prompt go quiet?
I wanted a tiny, reproducible way to find the blind spot instead of guessing. The answer is a needle-in-a-haystack test. This article is the complete case study, including the script I ran on a free server.
To run dozens of long-context requests without burning a student-sized API budget, I used MonkeyCode's operator-provided free tier: 30 million free tokens and a free server option.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Why a context window is not a guarantee
A model card can tell you a model accepts 128,000 tokens, but it cannot tell you whether the model actually uses the middle of that window. Real attention shifts with position, sequence length, and the presence of later text. When my notes lost a theorem, the model's context window was nowhere near full.
A haystack test makes the failure visible. You place one unique fact, the needle, at a controlled position inside a long filler text, then ask the model to retrieve the fact. Move the needle from start to end, measure the recall at each position, and look for the drop-off.
That one idea is enough for a useful student experiment. You are no longer asking "Can this model read long inputs?" You are asking "Where does this model stop retaining details?" The second question is much more practical.
Building the test
The core is intentionally small: generate a prompt, call an OpenAI-compatible endpoint, check whether the exact needle appears in the reply. I use standard Python plus requests, so the script runs on a free server without a heavy framework.
Here is the full probe:
import argparse
import json
import os
import random
import sys
import time
import uuid
from typing import Any
import requests
FILLER_WORDS = (
"token window memory context attention layer sequence position embedding "
"transformer retrieval needle haystack recall prompt history dialogue"
).split()
def make_prompt(total_tokens: int, needle: str, position_ratio: float) -> str:
# Approximate tokens as words; real models split differently, so this is a
# low-cost probe, not a precise scientific instrument.
needle_text = f" The special fact is {needle}. "
words: list[str] = []
current = 0
while len(words) < total_tokens:
if current == int(total_tokens * position_ratio):
words.append(needle_text)
current += len(needle_text.split())
else:
words.append(FILLER_WORDS[current % len(FILLER_WORDS)])
current += 1
# The question goes at the end so the model must search backwards.
return " ".join(words) + "\n\nQuestion: What is the special fact?"
def ask(model_url: str, model_name: str, api_key: str, prompt: str, temperature: float = 0.0) -> str:
response = requests.post(
f"{model_url}/chat/completions",
headers={"Authorization": f"Bearer {api_key}"},
json={
"model": model_name,
"messages": [{"role": "user", "content": prompt}],
"temperature": temperature,
},
timeout=180,
)
response.raise_for_status()
data = response.json()
return data["choices"][0]["message"]["content"]
def check_found(needle: str, answer: str) -> bool:
return needle in answer
def scan_positions(model_url: str, model_name: str, api_key: str, total_tokens: int, positions: list[float]) -> None:
for ratio in positions:
needle = str(uuid.uuid4())
prompt = make_prompt(total_tokens, needle, ratio)
try:
answer = ask(model_url, model_name, api_key, prompt)
found = check_found(needle, answer)
print(f"position={int(ratio*100):>3}% tokens={total_tokens} found={'yes' if found else 'NO'} answer_snippet={answer[:60]!r}")
except requests.exceptions.RequestException as exc:
print(f"position={int(ratio*100):>3}% request_error={exc}")
time.sleep(1.0) # be polite to shared free endpoints
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--model-url", default=os.environ.get("MODEL_URL", ""))
parser.add_argument("--model-name", default=os.environ.get("MODEL_NAME", ""))
parser.add_argument("--api-key", default=os.environ.get("API_KEY", "not-need-for-local"))
parser.add_argument("--total-tokens", type=int, default=4000)
parser.add_argument("--positions", type=float, nargs="+", default=[0.0, 0.25, 0.5, 0.75, 1.0])
args = parser.parse_args()
if not args.model_url:
# Local mock for offline validation.
print("Set MODEL_URL to your OpenAI-compatible endpoint. Running local mock instead.")
needle = "8f2e3e4e-9c2c-4f7c-af0d-1f9f16556c5f"
print(f"mock position=0% found=yes needle={needle}")
sys.exit(0)
scan_positions(args.model_url, args.model_name, args.api_key, args.total_tokens, args.positions)
What a typical scan looks like
After wiring up the endpoint, I ran the probe with a modest length first. The exact output will change from model to model, so treat this as an illustrative scan rather than a benchmark result.
$ export MODEL_URL=your_openai_compatible_endpoint
$ export MODEL_NAME=your_model_name
$ export API_KEY=your_key
$ python haystack_test.py --total-tokens 4000 --positions 0 0.25 0.5 0.75 1.0
position= 0% tokens=4000 found=yes answer_snippet='The special fact is 8f2e3e4e-...'
position= 25% tokens=4000 found=NO answer_snippet='I could not find a special fact.'
position= 50% tokens=4000 found=NO answer_snippet='The text does not mention a special fact.'
position= 75% tokens=4000 found=yes answer_snippet='The special fact is 3d6f...'
position=100% tokens=4000 found=yes answer_snippet='The special fact is 9c2c...'
That middle dip is the quiet zone. It matches the failure I saw with the lecture transcripts: content near the middle was the first to disappear. The scanner gives me a repeatable way to widen that zone across context lengths.
A useful next step is to run the same five positions with --total-tokens 8000, then 16000, then 32000. Watch whether the quiet zone grows, moves, or deepens. That tells you more than reading model-card context claims.
Common errors I hit
The first mistake was leaving temperature at its default. A higher temperature introduces variation that can make recall fail for no structural reason. Set it to zero.
My second mistake was using a natural needle phrase, only to discover it also appeared in the filler. Use a UUID; the probability of accidental matches is low and the verification is exact.
The third was sending a total that exceeded the endpoint's context limit and getting a JSON error. The probe does not prevent that, so check the model limit before scanning larger windows.
Finally, do not judge a model by one position. One miss at 40% is not a conclusion. You need the contrast across positions, and ideally two runs with different seeds, before a pattern becomes meaningful.
What I would do differently now
I would start with the mock branch before spending a single live token. The local mock validates that the script prints, checks, and exits correctly. That saved me from debugging long requests against a live endpoint.
I would also log every prompt and answer to a JSONL file instead of just printing. When you run several models or context sizes, the terminal output becomes hard to compare. A line like {"model":...,"total_tokens":...,"position":...,"found":...} is much easier to plot later.
Most importantly, I would use the free token budget for the repetition this experiment demands. One long scan is cheap; twenty scans across lengths and models is not. The 30 million free tokens removed that obstacle, and the free server kept the work off my laptop while I was in class.
Who should skip this method
A haystack test measures retrieval, not reasoning. If your real task is generating an argument from multiple sections, this probe only catches one failure mode among many. Do not use it as a general long-context benchmark.
Also skip this if you need production-grade document answers. There, chunking, retrieval, and citation are usually more reliable than hoping the model attends to a single massive prompt. The haystack probe is for understanding the blind spot, not for building around it.
The scanner itself is not a security boundary. It approximates tokens with words and can be fooled by models that rephrase the needle instead of echoing it exactly. Keep its limits in mind before drawing strong conclusions.
Copy the script, change total_tokens and positions, and post the sharpest contrast you find. I am curious where the quiet zone starts for your model.
MonkeyCode provides free models that can run this workflow.
Top comments (0)