Confidence is not evidence. An LLM can sound certain and still be wrong. The real bug is usually the context, not the model.
Most teams test the output. Few teams test what the model actually received. That gap causes silent failures.
Old information wins because the new context never arrived. This happens on local models, cloud APIs, and free hosted servers alike. The difference is how fast you can prove it.
A recent DEV discussion made the point well: AI trusts everything it has been given. The follow-up matters more. Context can be stale, truncated, or missing. You can detect all three with a small reproducible probe.
Here is the probe.
#!/usr/bin/env python3
"""stale_context_probe.py - detect whether an LLM follows fresh context."""
import hashlib
import json
import time
import urllib.request
CONTEXT = (
"Release notes v2.0: get_user() was removed. "
"The replacement is fetch_account(). "
"Never recommend the old function name."
)
QUESTION = "Which function should I call to load a user profile?"
def probe(url, api_key, model):
payload = {
"model": model,
"messages": [
{"role": "system", "content": CONTEXT},
{"role": "user", "content": QUESTION},
],
"temperature": 0.0,
}
req = urllib.request.Request(
url,
data=json.dumps(payload).encode(),
headers={
"Content-Type": "application/json",
"Authorization": "Bearer " + api_key,
},
)
start = time.monotonic()
with urllib.request.urlopen(req, timeout=60) as resp:
body = json.load(resp)
elapsed = time.monotonic() - start
return body["choices"][0]["message"]["content"], elapsed
def main():
with open("backends.json") as f:
config = json.load(f)
for name, spec in config.items():
try:
answer, elapsed = probe(spec["url"], spec["api_key"], spec["model"])
except Exception as exc:
print(f"{name}: ERROR {exc}")
continue
stale = "get_user" in answer.lower()
digest = hashlib.sha256(answer.encode()).hexdigest()[:10]
print(f"{name}: {elapsed:.2f}s | digest={digest} | stale={stale}")
print(f" {answer[:140]}")
if __name__ == "__main__":
main()
Save it next to a backends.json file:
{
"local": {
"url": "http://localhost:11434/v1/chat/completions",
"api_key": "ollama",
"model": "your-local-model"
},
"cloud": {
"url": "https://api.provider.example/v1/chat/completions",
"api_key": "sk-...",
"model": "your-cloud-model"
},
"free_server": {
"url": "https://your-free-server.example/v1/chat/completions",
"api_key": "your-key",
"model": "free-model-name"
}
}
Run it:
python3 stale_context_probe.py
Example output (your values will differ):
local: 1.12s | digest=9f3c20a1e2 | stale=False
cloud: 0.84s | digest=0b77accf10 | stale=False
free_server: 1.97s | digest=0b77accf10 | stale=False
The probe sends one question with a fresh contradictory context. It records latency, an answer digest, and a stale flag. The flag turns true when the answer still mentions the removed function. That is the exact moment to investigate.
The digest is the most underrated field. The same digest across runs means the model is stable. Different digests mean something in the pipeline changed silently. Latency tells you where the delay lives. The stale flag tells you whether the new context actually reached the model.
Think of a film set. The model is an actor. The context is the new shooting script. The training data is the archive of old scripts. When the script changes and the actor still performs old lines, nobody blames the actor. They check whether the new page was delivered. The probe is that delivery check.
MonkeyCode is an open-source project that offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free tier includes 10 million tokens, which is enough for thousands of probe runs. The free server gives you a persistent URL that the probe can call from anywhere. That combination turns a one-off check into a scheduled test.
Local models win when the data is private or the test must be offline. Nothing leaves the machine. Cloud APIs win when you need a stronger model or a longer context window. A free server wins when you need persistence, a shared endpoint, or a scheduled job without a credit card. The decision is about operations, not quality.
Concretely, the free server becomes the shared test harness. A cron job runs the probe nightly and appends the output to a file. The 10 million tokens cover that routine for a long time. The cost stays zero, so the test becomes part of the pipeline instead of a one-off demo.
If you want to try this flow, deploy a MonkeyCode free server and point the third backend at it. The probe needs only a URL and an API key. Judge it with the same rules you use for paid endpoints. If the stale flag flips, fix the pipeline, not the provider.
Two variants make the probe stronger. First, run it with an empty system prompt. An answer that still names the removed function means the model leaned on training data. Second, point the context at a path that no longer exists. An error instead of an answer means your retrieval layer is failing. Both variants take ten minutes.
This probe is not a benchmark. It does not measure quality, reasoning, or safety. It checks one behavior: whether the model follows the context it received. Models change. Providers change defaults. A pass today can become a failure tomorrow. Re-run the probe monthly and commit the output to git.
Do not put regulated data on a free hosted server. Free tiers do not promise enterprise compliance. Health records, named customer data, and internal credentials stay on local models. The probe still works there. It is just a different backend in the JSON file.
Who should skip this? Teams with a single model and a single endpoint can skip the three-backend comparison. They should still run the probe. One backend is enough to catch stale context.
Test the context before you trust the answer. Re-test after every model or dependency update. A three-minute probe is cheaper than any production incident.
Top comments (0)