Ask a language model how confident it is, and it will happily say "95%". The number sounds precise, but it often means nothing. For most chatbots that's harmless. For an on-call engineer deciding whether to restart production at 3 a.m., it's dangerous.
Our team built DejaVu, an AI incident agent that remembers how past outages were fixed. One rule guided a lot of our design: the confidence score must be earned from evidence, not invented by the model.
Where the evidence comes from
DejaVu stores every past incident in Hindsight, an open-source agent memory system by Vectorize. When a new error comes in, Hindsight recalls the most similar past incidents, and the model (GPT-OSS 120B on Groq) reasons over only those.
If "agent memory" is new to you, Vectorize's explainer on what agent memory is is a quick read. The key idea for us: memory gives the agent facts it can point to.
Rule 1: you can only cite what memory returned
The model must list the past incidents it used as evidence. But models are messy. Sometimes they cite an incident that was never recalled. Sometimes they write the id as "12", sometimes as "#12", sometimes they cite the same one twice.
So we clean the evidence before trusting it:
def clean_evidence(evidence, recalled_ids):
cleaned, seen = [], set()
for e in evidence or []:
raw = str(e.get("incident_id", "")).strip().lstrip("#")
if not raw.isdigit():
continue
iid = int(raw)
if iid in recalled_ids and iid not in seen: # must come from memory, once
seen.add(iid)
cleaned.append({**e, "incident_id": iid})
return cleaned
Anything the model made up is dropped. A duplicate citation can't double-count.
Rule 2: confidence comes from proven fixes
Next, we count how many of the cited incidents had a fix that actually worked, and map that onto a fixed scale:
| Proven matches in memory | Confidence |
|---|---|
| 3 or more | 90 to 98% |
| 2 | 75 to 89% |
| 1 | 55 to 74% |
| none | 30% or less |
proven = sum(1 for c in cited if c["fix_successful"])
if proven >= 3:
conf = min(98, max(model, 88 + 2 * proven))
elif proven == 2:
conf = min(max(model, 75), 89)
elif proven == 1:
conf = min(max(model, 55), 74)
else:
conf = min(model, 30)
The model's own number still matters inside each band, but it can never escape the band. With no proof, the agent can't claim more than 30%, and it says plainly: "No matching incident in memory."
Under the score, the app shows the reason in words, like "5 of 5 matching past incidents were fixed this way." Engineers don't have to trust a number. They can check the incidents.
The bug our tests caught
We wrote an automated test suite with pytest, with Hindsight mocked out so tests run in about a second. Part of it was written by an AI coding agent working in the Band desktop app.
One test fed the calibration a model that claimed 100% confidence with 5 proven matches. Our old formula let that through. A score of 100% breaks the whole idea, because nothing in incident response is certain. The fix was a hard cap at 98.
Later tests guard against the messy-model cases above: ids written as text, duplicate citations, broken JSON replies, and fix steps returned as one long string. The suite is now 56 tests.
Why this matters
- A confidence score is a promise. If it's wrong once, engineers stop trusting it.
- Ground it in evidence you can show. Memory makes that possible, because every claim links to a real past incident.
- Be honest when you don't know. A low score with "nothing similar in memory" is more useful than a confident guess.
- Test the edge cases. The bug we found would never show up in a happy-path demo.
You can read more about how recall works in the Hindsight docs.
What it looks like in practice
Here's how the same rules play out on two different alerts.
A repeat outage. A payments service throws KeyError: 'amount' after a deploy. Hindsight recalls 5 past incidents with the same cache bug, and all 5 fixes worked. The score lands at 98%, with the reason "5 of 5 matching past incidents were fixed this way." The engineer can open each card and see the service, the error and the time it took.
A brand-new problem. An error we had never seen before comes in. Recall still returns something, because it always returns the closest matches, but none of them were really fixed this way. The cleaned evidence is thin, nothing is proven, and the score stays at 30% or less. The screen says there's no matching incident in memory, so the engineer knows to investigate rather than follow a guess.
Same code, same model, very different confidence, and both are honest.
Try it
- Code (including the tests): https://github.com/Pratheeksh-7S/dejavu-incident-agent
- Demo video: https://youtu.be/Nu_sktguGMk
- Hindsight: https://github.com/vectorize-io/hindsight
An AI that says "I'm not sure" when it isn't sure is one an engineer can actually rely on.
Top comments (0)