Every "AI reviews your tickets" demo has the same hole in it: the output reads well whether or not it is true. Plausibility is the one thing a language model is reliably good at, so grading on it measures nothing.
I wanted a number instead. Not "do these risks sound smart" but did it name the thing that actually broke — and the only way to answer that is to know what broke.
Borrowing an answer key from history
Open-source trackers write the answer down for you. Somewhere in a big repository there is always an issue that says "regression introduced in #39177". That sentence is a free, retrospective answer key: it tells you what went wrong, and it tells you which change caused it.
So the method was:
- Search five large repositories (Airflow, Elasticsearch, Prometheus and two others) for issues whose body says introduced in #N, caused by #N, regression from #N, broken by #N or after PR #N, filed between January 2023 and August 2025.
- Take the change description — the author's own words, boilerplate stripped — as the ticket.
- Assess it, blind, with no knowledge of the follow-up issue.
- Ask one question with a real answer: did the assessment name the failure the follow-up issue describes?
423 candidates, 17 with an extractable reference, 6 kept for range: two permission failures, one string-length limit, one URL construction, one metrics-consistency bug, and one deliberately boring-sounding refactor ("tidy up ImmutableOpenMap and Iterators a bit") that turned out to break S3 deletions.
The model was gpt-5.5 — a frontier model, not a cheap one. That matters, because it removes "use a better model" from the list of available excuses for the misses.
The score
Two hits, two near-misses, two misses out of six.
That is the honest, blind number. A later round scored three hits, but one of those cases had stopped being blind — I had written a prompt line after watching it fail, so it cannot be counted as evidence again. Round two shows the fix worked; it does not show the product is better than round one measured.
What a hit looks like, from the Airflow case where the BigQuery operator started failing on long task IDs:
the ticket said "set dag ID and task ID as BigQuery job labels"; the assessment asked what happens when an ID exceeds the label length the API accepts
The follow-up issue, two months later: operator fails for task IDs of 64 characters.
What a miss looks like: the "tidy up" refactor. The assessment produced sensible, generic refactoring risks. The actual failure — an access-control exception when deleting files in S3 buckets, because an iterator change moved code into a different security context — was not among them. Nothing in the ticket text hinted at it. A human reviewer with no knowledge of that codebase would have missed it too, which is the point: the ceiling here is the information in the ticket, not the model.
The experiment that went against me
Each risk came with a severity, occurrence and detection rating, and I computed a priority from them — the automotive-industry Action Priority lookup, chosen because multiplying ordinal ratings (the classic FMEA RPN) is mathematically meaningless.
The obvious next step is to sort the list by that priority. So I measured it, on five held-out tickets whose real failure was known, scoring where the risk that actually came true ended up in the list:
| ordering | rank of the real failure, per ticket | mean |
|---|---|---|
| the model's own order | 1, 1, 2, 1, 1 | 1.2 |
| likelihood, then impact | 2, 4, 1, 2, 3 | 2.4 |
| severity, then the model's order | 3, 5, 3, 1, 2 | 2.8 |
| computed Action Priority | 3, 7, 3, 1, 2 | 3.2 |
Every sort I applied made it worse. The model writes the risk that matters first, and every reordering on top of its own ratings threw that signal away.
Five tickets is not proof, and this is the first thing I would re-measure on fresh ones. But the data I have says: keep the ratings, show them, and do not sort by them. The feature I was most pleased with was the one the measurement told me to switch off.
There is a second, related finding. Asked for likelihood and impact on a low/medium/high scale, the model answered "medium" for 78% of 96 risks, including all five that came true. A list sorted on that is sorted on nothing. Replacing it with three anchored 1–5 ratings — what has to be true for this to happen, the worst realistic consequence, how late anyone would notice — produced numbers that at least vary. Whether they are calibrated is a different question, and the only thing that can answer it is outcomes recorded over time.
Silence is a feature, and it has a number too
The failure mode nobody demos is the one that kills these tools in practice: risks on changes that are fine. A tool that produces eight paragraphs of concern about a typo fix teaches its users to skim, and after a week they skim the real ones too.
So the suite has negative controls — real changes that did not break anything, including a genuinely additive feature. The target is not "few risks", it is nothing at all:
- benign change: 0 risks, 27 output tokens
- real, additive feature in the same run: 8 risks, 1 605 output tokens
Getting that rule in cost one paragraph in the prompt and removed an entire category of noise. It is the only change in six rounds that told the model to produce less, and it bought more than the three before it, all of which told it to look harder.
What I would tell you if you are building something similar
- Find an answer key before you build the grader. Trackers, incident reports, post-mortems and reverted commits are all retrospective ground truth, and they are free.
- Grade on a question with a real answer. "Did it name what broke" beats any rubric about helpfulness.
- Measure your sorting. Ordering is a model you did not know you were building, and it can be worse than no ordering at all.
- Put benign inputs in the suite. If silence is not measured, it will not survive the next prompt change.
- Publish the misses. Two of six is a number you can improve. "It writes great risks" is not.
The fixtures, the answer keys and every number above are public: https://github.com/mistereshka/premortem-eval
The tool this came out of is a Jira app called Premortem — it assesses an issue before work starts, and the pull request when it arrives, on your own AI key. It is free on the Atlassian Marketplace: https://marketplace.atlassian.com/apps/3455340266/premortem-risk-assessment-for-jira
The prompt is not in the repository, and I would rather say so than pretend the evaluation is fully reproducible: what is published is enough to check the claims, not enough to rebuild the tool.
Top comments (0)