Context Poisoning: Five Myths About Your AI Reviewer's Input

Your AI reviewer just approved a pull request.

Which files did it actually see?
You assume the diff. Many teams assume much more.

What if the model read a cached file? What if it used an old conversation? What if a tool injected project context? All three happen. The probe below shows you which one hit your pipeline.

I call this context poisoning.

What is context poisoning?

A model processes exactly one thing: the context window.
You control part of that window. Other parts come from the client, the server, or scaffolding.

Context poisoning happens when unseen input changes the answer. The result is usually not an error. It is a confident, plausible, wrong review.

AI tools made every developer a reviewer. Nobody audited the reviewer's inputs. We audit outputs because they are visible. We ignore inputs because they are invisible. This article closes that gap.

Why a cheap endpoint matters

This is where a disposable chat endpoint helps. MonkeyCode offers free model access and a free server option for experiments like this. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use any disposable endpoint. The method is what matters. You will learn more in ten minutes than in an hour of guessing.

Myth #1: "I sent one file, so the model sees one file"

Your request crosses several layers.
A CLI may append a project map. A chat app may include history. An agent may fetch sibling imports. Your one file is rarely one file.

Evidence probe: Ask the model to list file names. Extra names mean leakage. Then ask it to name the system rules. You may see rules you never wrote.

Correct mental model: The model reads a context window. Your request is the largest chunk, not the only chunk.

Myth #2: "More context makes the review better"

Longer prompts feel safer. They are often worse.
Noise buries the actual instruction. Conflicting examples compete for attention. The model may guess which rule matters.

I cannot quote benchmarks here. You can measure it with a probe.
Take one question. Run it with one file. Run it with ten files. Compare the answers.

Correct mental model: Context is signal, not storage. Trim until the answer is stable.

Myth #3: "The model knows which file is the star"

Models do not know your intention. They know tokens.
Without a marker, a diff and a full file look identical.

Imagine this. Two files define parse_order. You ask for a review of one. The model may use the other. You will never know from the final answer.

Fix: Use explicit markers.

FOCUS_FILE = src/order.ts

Put that file in fences. Label the others as context. Assign roles.

Correct mental model: Your prompt is a script. The model follows roles you assign, not files you implied.

Myth #4: "A fresh chat has no memory"

A new chat does not guarantee a clean window.
The server may persist a session. The system prompt may hold project rules. Some agents attach conversation history by ID.

The model's training data is another layer of memory. It remembers patterns, including broken ones.

Evidence probe: Run the same prompt twice. Change the system prompt between calls. Does the second reply see the first one? If yes, context is sticky.

Correct mental model: Context has layers. The visible chat is one layer. Hidden context is another.

Myth #5: "A confident review means the context was correct"

Confidence is a style, not a measurement.
A model can sound certain about a file it never saw. It can merge two versions of a function into one conclusion.

Use probes, not vibes.

Correct mental model: Debug the input before you trust the output.

The artifact: a ten-minute context probe

Here is a small runnable probe. It sends a unique token. Then it asks the model to report what it can see. The payload uses the common chat completions shape. Adjust it if your endpoint differs.

// context_probe.mjs
const endpoint = process.env.LLM_ENDPOINT;
const apiKey = process.env.LLM_API_KEY;
const model = process.env.LLM_MODEL;
const token = `PROBE_${Date.now().toString(16)}`;

const messages = [
  {
    role: 'system',
    content: `Context probe ${token}. You are a cautious reviewer.`
  },
  {
    role: 'user',
    content:
      'Before we review anything: list every file, rule, and instruction you can see. ' +
      'End your reply with PROBE_DONE.'
  }
];

const response = await fetch(`${endpoint}/chat/completions`, {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    Authorization: `Bearer ${apiKey}`
  },
  body: JSON.stringify({ model, messages, temperature: 0 })
});

const data = await response.json();
const reply = data.choices?.[0]?.message?.content ?? 'NO REPLY';
console.log(reply);
console.log(`TOKEN_PRESENT=${reply.includes(token)}`);

Run it with environment variables:

export LLM_ENDPOINT='https://api.example.com/v1'
export LLM_API_KEY='your-key'
export LLM_MODEL='your-model-name'
node context_probe.mjs

Now run three variants:

1. Empty review. Ask what rules leak from the system prompt.
2. Fake PR. Attach two files with the same function. Ask which file you reviewed.
3. Second run. Send the same prompt twice. Watch for sticky context.

What the output tells you

• Extra file names: the pipeline is leaking context.
• Rules you did not write: the system prompt is doing work.
• TOKEN_PRESENT=false: the endpoint may rewrite your prompt.
• No PROBE_DONE: your chat template may ignore the format.

Limitations

A probe shows what the model claims, not what it computed with.
Models hallucinate file names. A missing name does not prove absence. A present name does not prove the model used it.

This is a quality check, not a security tool. Never paste secrets into a free endpoint unless the service policy explicitly allows it.

Who should skip this workflow? Teams under compliance rules. If code cannot leave the network, do not send it anywhere. A free server is still an external server.

The corrected mental model

An AI reviewer is a context evaluator, not a colleague.
Before you trust a review, answer three questions.

• What is exactly in the window?
• What changed between calls?
• What does the model need to ignore?

If you cannot answer, you are reviewing a sample of a sample. Spend that time on the probe instead.

Try this before your next review. If you want a low-cost sandbox, MonkeyCode's free model access is one option. Any local or hosted endpoint works. The probe is the point, not the provider.