My RAG gave different answers to the same question. One setting I never set, one I never checked

TL;DR: Non-deterministic RAG answers with no error came from two things: temperature and seed left to the provider's defaults, and OpenRouter's context-compression plugin cutting the middle of long prompts. I found it when the same extraction, run twice, returned two different answers. A small experiment with a fictional company reproduces both. Sampling changed how the answers were worded. The cut changed whether the model could answer at all.

I ran the same extraction twice on the same input and got different results.

That bothered me more than an obvious error would have. Both runs completed successfully, with no exception and no warning. Just two answers that didn't agree.

Before changing the prompt, the chunking or the retrieval, I wanted to know what the model was actually receiving. I found two things in my own setup that I had overlooked.

Non-deterministic RAG answers: temperature and seed I never set

The first one was something I had never configured. I hadn't set temperature or seed, so the provider's defaults decided how the model sampled. Two identical requests can come back different, and the defaults can change without notice.

I now set both explicitly, in one place, on every call. That has limits. Even with both set, most providers don't promise bit-for-bit identical output: batching and floating-point order on the GPU still introduce small differences. Some providers don't let you set them at all. On OpenRouter, seed only works where the underlying provider supports it. The Anthropic API has no seed, and on API version 2026-10-21 a request that sends temperature, top_p or top_k returns a 400 on every model, even at the default value. Claude 4.7 and later models already accepted only the defaults before that. On those models you can't turn run-to-run variance off, so the pipeline has to tolerate it.

OpenRouter's context-compression plugin silently cuts the middle of the prompt

The second one was worse. OpenRouter's context compression plugin (plugins: [{ id: "context-compression" }]) was hardcoded into every call, and I had never checked what it did. When a prompt is longer than the context window, it tells the provider to shorten it by removing text from the middle.

You can hit this without configuring anything. According to OpenRouter's docs, it is on by default for every endpoint with a context of 8,192 tokens or less. To turn it off, send { "id": "context-compression", "enabled": false }.

The request still succeeds. The model answers from the start and the end of the prompt, and nothing in the response says the middle was cut. That is also one way "same input" becomes "different prompt": if retrieval breaks score ties in no fixed order, or chunks fetched in parallel are appended as they arrive, the prompt changes from run to run while the question and the documents stay the same. With a middle cut, a different order means different evidence survives.

Reproducing the evidence loss: 8K middle cut vs. full context

I wanted to see both effects separately and without client data, so I generated a report for a fictional company: about 9,300 tokens of monthly operations, branch reports and customer accounts. The three facts that matter (partners and shares, total debt, 2025 revenue) sit at 44% to 45% of the text, inside the band a middle cut removes. I asked three questions: who the partners are, how much the company owes, and what it billed in 2025.

The cleanest example came from the cut runs. I asked the revenue question with the chunks in two different orders. One run came back "BRL 38.4 million". The other came back "NOT FOUND". Same report, same question, no error.

I used google/gemini-3.1-flash-lite through OpenRouter. Gemini has a 1M context, so compression never triggers there. For the cut runs I removed chunks from the middle in my own code, the way the plugin does, until the prompt fit 8K. The reordered rows add up to 9 calls: three questions, one run in the first order and two in the second. In total, 42 calls were sent. The refused row sent none. The last row uses the real plugin on an 8K model. The simulated cut shows what happens when evidence is lost. Only that last row exercises OpenRouter's actual mechanism, and it is three calls on one model. I didn't pin the sub-provider either. The logs show that Google served every Gemini call and NextBit served every Gemma call, so routing didn't mix providers in these runs. The code, the report generator and every answer are in this gist.

Setup	Model	Calls	Correct
No report at all (control)	Gemini 3.1 Flash Lite	3	0, all NOT FOUND
Full report, provider defaults	Gemini 3.1 Flash Lite	9	9
Full report, temperature: 0 + seed: 42	Gemini 3.1 Flash Lite	9	9
Simulated middle cut to fit 8K	Gemini 3.1 Flash Lite	6	0
Simulated cut, chunks in two other orders	Gemini 3.1 Flash Lite	9	3
Doesn't fit, call refused before sending	none	0 (3 refused)	none sent
Own retrieval, trimmed to fit 8K	Gemini 3.1 Flash Lite	3	3
Real OpenRouter plugin, full report	google/gemma-2-27b-it (8K)	3	0

With the provider defaults every answer was right, but the partners question came back in three different phrasings over three runs. With temperature: 0 and a seed, each question came back word for word the same every time. A step that parses or diffs the output would see the first case as three different results.

With the middle cut, every answer was NOT FOUND. The system prompt told the model to say that when the context lacks the answer, and the instruction held. Chunk order then decided which facts survived: in one order the revenue came back right and the other two stayed lost, in another the partners came back and the revenue was gone.

The real plugin cut without a word. On the 8K model, OpenRouter reported about 2,500 prompt tokens for the report, with no error and no warning, and all three answers were NOT FOUND. Refusing the call when the prompt didn't fit sent nothing, and picking the chunks with my own rule, trimmed to fit, got all three right.

NOT FOUND is the good version of this failure, and it only helps if something counts it. A pipeline that stores the answer and moves on records the gap as if it were data.

The bad version showed up when I ran the same test on a public document the model already knew, the Apollo 11 article on Wikipedia. With the real plugin, the article reached the model as 2,173 tokens, again with no error. Asked for the early-planning names of the two spacecraft, the model answered "Columbia and Eagle". Those are the final names, not the early ones (Snowcone and Haystack). The answer reads fine and is wrong.

What the runs showed, and what they didn't

The runs showed that:

• sampling settings changed the wording of the answers, not the facts;
• cutting the middle of the prompt removed the evidence, and every answer became NOT FOUND;
• chunk order decided which facts survived the cut;
• the real plugin, on an 8K model, cut the report to about 2,500 tokens with no error or warning.

They didn't show that:

• these two causes explain every disagreement in my original pipeline. They reproduce the symptom, and I can't rule out other causes there;
• this happens often in production. It is a small run: two documents, one model for the cut runs, three calls or fewer per setup in most rows;
• my retrieval rule works beyond three questions on one report;
• a large context window solves it. With 1M tokens there was no cut, but at 9,300 tokens I didn't test whether the model still uses evidence from the middle of a much longer prompt;
• the model filled the Apollo gap from training. That is my guess, and I didn't test it.

How to make a RAG call fail when the prompt doesn't fit

A minimal version, written for this post. Count the tokens before sending, refuse the call if prompt plus output doesn't fit, turn compression off so the provider can't cut anything on its own, pin the host, and check the count the provider reports after the call:

class ContextOverflowError extends Error {}
class SilentTruncationError extends Error {}

const MODEL = "vendor/model-name";  // exact model id, never a "latest" alias
const CONTEXT_LIMIT = 32_768;        // from the model's page, not a guess
const MAX_OUTPUT = 1_024;
const SAFETY_MARGIN = 0.05;          // your tokenizer and the provider's can disagree

function assertFits(promptTokens: number) {
  const budget = Math.floor(CONTEXT_LIMIT * (1 - SAFETY_MARGIN)) - MAX_OUTPUT;
  if (promptTokens > budget) {
    throw new ContextOverflowError(
      `prompt has ${promptTokens} tokens, budget is ${budget}`
    );
  }
}

assertFits(countTokens(messages));   // countTokens: your own tokenizer, not defined here

const body = {
  model: MODEL,
  messages,
  temperature: 0,                    // drop it on Claude: the API rejects it
  // seed: 42,                       // only if the provider supports it
  max_tokens: MAX_OUTPUT,
  plugins: [{ id: "context-compression", enabled: false }],
  provider: { only: ["provider-slug"], allow_fallbacks: false }, // same model can run on different hosts
};

const res = await callModel(body);
if (res.usage.prompt_tokens < expectedTokens * 0.8) { // much lower than sent: something cut the prompt
  throw new SilentTruncationError(
    `sent ~${expectedTokens} tokens, provider counted ${res.usage.prompt_tokens}`
  );
}

Token counts are estimates unless you measure them with the provider's own tokenizer. The margin lowers the risk without removing it, which is why the check after the call matters. The provider's count can run a little above yours, because chat templates add tokens. A count far below yours means the prompt was cut. In my runs the report went from about 9,300 tokens to about 2,500, a drop no tokenizer difference explains. Pick the threshold from your own logs; 0.8 is a placeholder.

If the prompt is too long, I decide what to drop in my own code, with a rule I can read and test. I also tell the model to answer NOT FOUND when the context lacks the answer, and I count how often it does. I pin the exact model version and the host, and record which ones answered. OpenRouter can serve the same model name from different providers, sometimes at a different quantization, unless you restrict it with provider.only or provider.quantizations.

How I debug RAG now

I used to spend my time on retrieval whenever a RAG answer looked inconsistent. I still think retrieval matters, but this experiment changed the order I work in.

Now I look at the exact request first: the sampling settings, the context budget, and whether the evidence made it into the prompt. Only after that do I change chunk sizes or prompts.

To make that comparison possible, I sort the chunks in a fixed order before building the prompt (score, then document id, then chunk index) and log a hash of the final prompt next to the answer. Two runs with the same hash sent the same thing.

What still bothers me is how ordinary the failed responses looked. The requests succeeded and the answers read like normal answers. For document extraction I want the system to catch that before anyone reads the result.

The next time two runs disagree, compare the two prompt hashes before you touch retrieval. If they differ, find where the difference came from. If they're identical, retrieval is cleared, and the next place to look is what happens after the request: sampling, and anything the provider does to the prompt.debugging,