Most of the reliability work around AI agents right now goes into the model and the retrieval. We tune the prompt, we improve the ranker, we add a re-ranker on top of that. What gets far less attention is a simpler question that sits one step later in the pipeline: by the time a piece of context actually reaches the model, is it still worth acting on?
That question is what I have been building around, and this is a short, honest account of why I think it matters.
The gap nobody names
An agent acts on whatever context reaches it. If that context is stale, if it cannot be traced to a source, or if it contradicts itself, the agent usually acts on it anyway. The failure is quiet. There is no exception, no red line in a log, nothing that says the system relied on something it should not have. In a low-stakes workflow that is a nuisance. In a regulated or high-value one, that missing record is a real exposure, because afterwards there is no way to show that the context was ever checked.
Retrieval scoring does not close this gap. A ranked list tells you which chunks are most similar to the query. It does not tell you whether the top chunk is three days out of date, whether it can be attributed to anything, or whether it agrees with the chunk below it. Similarity and integrity are different questions.
What context integrity means
I use three properties, because they are the ones that actually break in practice:
- Fresh. Has the underlying source changed since this context was retrieved? Freshness is not one global timer. A price decays in minutes, a definition in years, so the model has to account for the source, not just the timestamp.
- Attributable. Can this context be traced back to a source you can name? Unattributable context is the kind that quietly becomes a hallucination once the model repeats it.
- Coherent. Does the context hold together, or does it carry internal contradictions that the model will average into something confidently wrong?
A piece of context can score high on relevance and fail all three. That is the case worth catching.
A verdict you can verify without trusting me
The part I care about most is not the score. It is that the result is independently checkable.
Every evaluation is written to a ledger and signed with an Ed25519 key. Anyone can take a verdict and verify it offline: no account, no API key, no call back to my server. That property is deliberate. A trust layer that you have to trust on faith is not a trust layer. If I tell you the context was checked, you should be able to confirm that claim yourself, later, without me in the loop.
This is the difference between a promise and proof. A promise is a sentence in a pitch. Proof is a signature you can verify after I am gone.
What is built, and what is not
I would rather be precise than impressive, so plainly:
Built and running today: the evaluation engine that scores freshness, attribution, and coherence with a decay-adjusted model; signed, ledger-backed, offline-verifiable verdicts; an interface over the Model Context Protocol; and a hosted endpoint with a live before-and-after demo. The specification is published under an MIT licence.
Not built yet, on the roadmap: a pass / warn / refresh / block safety harness, and a control-plane dashboard with telemetry. I mention them so the picture is honest, not to imply they exist.
See it rather than take my word
There is a before-and-after demo here: https://freshcontext.dev/context-integrity-demo
Run something through it, then verify the verdict yourself. If you build agents, retrieval, or memory systems and this problem is one you have hit, I would value hearing how it shows up for you. That is the part I am still learning.
— Immanuel Gabriel, Founder, FreshContext
Top comments (4)
The signature makes the verdict portable, but only if it commits to the exact context bytes or digest, source identity and version, policy version, and clock inputs used for freshness. Otherwise it proves who issued a conclusion without making that conclusion reproducible. This is close to the evidence-boundary work I’m exploring in agent-inspect. Does offline verification re-evaluate the rules from bundled evidence, or only verify the signed verdict?
The gap you are naming is real, and part of why it stays unnamed is that it has no error type. Stale, untraceable and self-contradictory context all produce a confident answer, and none of them raises anything a monitor can catch.
My version of the fix has been to make the context set a first-class object rather than a transient prompt fragment: every chunk carries a source id, a version and a retrieved-at timestamp, and the answer records which of them it actually used. That gives you the after-the-fact record, and it also makes contradiction mechanically checkable, since two chunks disagreeing on the same field is only detectable once they are structured.
Freshness is the property I would gate hardest, because it is the only one that degrades on its own with nobody touching the system.
The three properties are the right decomposition, but coherence is the one I think is hardest to make actionable in a multi-agent setup specifically. Two chunks can each individually pass fresh + attributable and still disagree, because they came from two different agents that each did their own retrieval at slightly different times against sources that hadn't converged yet — not stale data, just two true-at-the-time snapshots that no longer agree. Detecting that requires comparing across contexts pulled by different callers, not just scoring one context set in isolation, which seems like a much bigger surface than the per-chunk checks. Does your ledger track enough to catch contradiction across two separate evaluation calls, or is coherence currently scoped to contradictions within a single retrieved set?
"Similarity and integrity are different questions" — that's the line that names exactly what retrieval scoring misses.
The freshness framing is what gets it right. A global staleness timer doesn't work because a price decays in minutes and a definition decays in years. The decay model needs to be source-aware, not timestamp-aware. That's a constraint most RAG implementations don't even surface as a problem to solve.
The offline-verifiable Ed25519 signature piece is the design decision that matters most long-term. A trust layer you have to trust on faith is not a trust layer. The distinction between a promise and proof being a signature you can verify after I'm gone is the exact framing that separates a real integrity layer from a marketing claim. That property scales; the promise doesn't.