Part 2 of a series on auto-grooming Jira backlogs with ML and LLMs. Part 1 article: How I Auto-Groomed 500 Jira Tickets with ML and LLM for the original TF-IDF + Gemini pipeline this builds on.
The Question Part 1 Left Open
Quick recap for anyone who missed Part 1: I pulled 500 real Apache ZooKeeper tickets from Jira's public API, cleaned the text, ran it through TF-IDF and K-Means to get five clusters, then flagged likely duplicates with cosine similarity. That's the part any ML pipeline can do. The interesting part was Phase II — feeding those raw clusters and duplicate pairs into Gemini 2.5 Flash, grounded with three selectively-weighted RAG files (project context, release notes, and a 2026 roadmap theme), so the output came back as named clusters with PM recommendations, age-aware duplicate calls, and an executive summary instead of a spreadsheet of keywords.
But I built that pipeline with exactly one clustering method and one LLM, and never checked it against anything. If Gemini's cluster names sounded sharp, was that because Gemini did a good job, or because there was nothing else in the room to compare it to?
That's the question this part tries to answer. Instead of one pipeline feeding one LLM, I ran two clustering pipelines through two LLMs, producing twelve outputs total, and had a third LLM score all of them against a rubric. Then I scored the same twelve outputs myself, by hand, and compared.
I expected the comparison to tell me something about Claude versus GPT. It ended up telling me something more useful: about where an LLM judge can and can't be trusted.
Why Two Pipelines, Two LLMs
One pipeline gives you no baseline. Adding a second clustering method and a second model lets me vary two things independently:
- Clustering approach — sentence embeddings versus TF-IDF. Does semantic embedding actually help on a backlog this keyword-dense, or is TF-IDF's simplicity good enough?
- The LLM itself — Claude Sonnet 4.5 versus GPT-5.5, given identical inputs and identical RAG context.
More moving parts than Part 1, but it's the only way to make a claim like "Pipeline A clusters better" or "Claude reasons better about stale tickets" actually mean something, instead of being one anecdote dressed up as a finding.
How It's Wired Together
Five stages, and the diagram below the title maps them out, but here's the short version:
)
1. Two pipelines, same input. The same 500-ticket CSV runs through Pipeline A (sentence embeddings → K-Means → cosine similarity) and Pipeline B (TF-IDF → K-Means → cosine similarity), independently. Both pipelines picked K the same way — elbow method plus Silhouette Score, rather than reusing Part 1's fixed K=5 outright — so cluster count wasn't a hidden thumb on the scale for either method. Each pipeline spits out its own cluster set and duplicate list.
2. Both LLMs see both pipelines. This is the part that changes from Part 1. Claude Sonnet 4.5 enhances Pipeline A's output, then Pipeline B's. GPT-5.5 does the same — both pipelines, not just a matched one. Each of those four combinations produces the same three artifacts as Part 1: enhanced cluster names and descriptions, an enhanced duplicate list, and a PM Insight Report, grounded in the same three-tier RAG setup (project context, release notes, 2026 theme) and the same age categorization from Part 1.
Two LLMs, two pipelines, three artifacts each — twelve outputs. Having each LLM touch both pipelines matters: it lets me separate "the LLM writes a better PM report" from "the embedding pipeline handed it better raw material," which a matched design (Claude only sees Pipeline A, GPT only sees Pipeline B) would have muddied.
3. A neutral judge scores all twelve. Gemini 3.1 Pro — outside both the Claude and GPT families — scores every output against five criteria: Strategic Alignment, PM Recommendation Specificity, Age-Aware Duplicate Reasoning, Cluster Coherence, and Data Synthesis & Quality. Not every criterion applies to every output type (a duplicate list doesn't get scored on cluster coherence, obviously), so each output is scored out of whatever denominator actually applies. The rubric also asks Gemini to flag suspected hallucinations directly, and to score only what's explicitly present rather than inferring intent — a rule that turns out to matter a lot, below.
I'd considered having Claude and GPT also cross-judge each other's outputs, to control for a model favoring its own family. I skipped it here — with only twelve outputs, that's a different (and noisier) experiment than the one I actually wanted to run, which is simpler: does a judge that has no stake in either model agree with a human at all?
4. I reviewed all twelve myself, scoring against the same rubric, before comparing notes with the judge.
5. My review, not the judge's raw score, is what actually determined the final copy for any downstream use of this pipeline.
What the Comparison Actually Showed
The headline: I scored both models' work noticeably higher than Gemini did.
| Gemini avg | My avg | Gap | |
|---|---|---|---|
| Claude Sonnet 4.5 outputs | 48.9% | 70.5% | +21.6 pts |
| GPT-5.5 outputs | 47.8% | 63.9% | +16.1 pts |
That gap wasn't spread evenly. Breaking it out by criterion:
| Criterion | Avg gap (mine − Gemini's) |
|---|---|
| Data Synthesis & Quality | +1.17 |
| Age-Aware Duplicate Reasoning | +1.13 |
| PM Rec. Specificity | +0.88 |
| Strategic Alignment | +0.58 |
| Cluster Coherence | +0.50 |
The two biggest gaps — Data Synthesis and Age-Aware Reasoning — are also the two criteria that ask a model to do something with the data rather than just cite it: run arithmetic across ticket counts, or reason about what a ticket's age actually implies. That's not a coincidence.
The clearest example of this showed up in GPT-5.5's PM Insight Report from the embedding pipeline. Gemini flagged it as a hallucination, pointing to specific figures — a claim like "232 issues are Bugs and 147 are Improvements," a "63–121 issues per cluster" range — that it couldn't find written anywhere in the source documents. It scored Data Synthesis a 1.
I gave that same output a 4 on Data Synthesis. The numbers weren't invented — GPT-5.5 had counted and aggregated them from the actual ticket data it was working with. It was correct analysis that happened to produce specific-sounding figures, and because those figures weren't quoted verbatim anywhere in the RAG files, the judge read "not explicitly sourced" as "fabricated."
That's the finding I'd flag to anyone building something similar: an LLM judge, when it can't trace a number back to a quote, seems to default to calling it a hallucination — even when the number is a legitimate computation over data it actually had access to. From the judge's seat, a correct inference and a real hallucination can look identical unless it's willing to redo the math itself.
Worth being honest about the exception, too: on GPT-5.5's TF-IDF PM Report, the pattern flips — Gemini scored Strategic Alignment and Data Synthesis higher than I did. So this isn't "the judge is stingy" across the board. It's specifically the inference-versus-hallucination boundary where the disagreement lives, and it doesn't run one direction consistently enough to just apply a blanket correction factor.
(Full per-criterion scores for both models are in the Github evaluations folder if you want the raw numbers.)
A Note on the Judge Prompt
I won't reproduce the full judge prompt here — it's a fairly long, rigid rubric document, and it's more useful sitting in the repo where you can actually copy and adapt it than pasted into an article. The short version: it told Gemini to act as an impartial judge, gave it explicit 5/3/1 scoring anchors for each of the five criteria, specified which criteria applied to which output type, and included one rule that turned out to matter more than I expected — score only what's explicitly present, don't infer intent. That instruction is almost certainly why the judge treated GPT-5.5's derived figures as unsupported rather than derived: it was doing exactly what it was told, just a little too literally.
Delta, Not Kappa
Cohen's Kappa is the standard tool for this kind of comparison, and I looked at it first. But Kappa needs enough trials to be stable, and with six outputs per model on a 1–5 scale, it would've produced a number I didn't trust. A simple delta — my score minus Gemini's, per criterion — was less elegant but told me exactly where the disagreement was concentrated, which is what actually mattered here.
Where the Human Still Earns Their Keep
The GPT-5.5 hallucination flag above is the whole argument for keeping a person in the loop. It wasn't a vague "I felt more generous" adjustment — it was a specific, checkable disagreement about whether a number was invented or computed. A rubric and a capable judge get you most of the way there, but not all the way; they're not yet reliable at telling "unsupported" from "correct math the judge didn't bother to verify."
What I'd Do Differently Next Time
A few honest takeaways, in no particular order:
The judge is a triage tool, not a verdict. Gemini's scores tracked mine reasonably well most of the time — good enough to flag likely problems fast. Not good enough to be the last word, especially on hallucination calls.
The disagreement wasn't random — it clustered on the two criteria that require inference rather than lookup. That's a fixable design problem: give the judge a way to re-derive a number before flagging it, rather than pattern-matching against the source text alone.
Cross-pipeline coverage was worth the extra setup. Letting each LLM touch both pipelines is the only reason I can say anything about the LLM and the pipeline separately, instead of one tangled result.
What's Next
I want to try giving the judge model an actual calculator or code tool, so it can check a number before calling it fabricated. I'd also like to run this on a bigger, messier backlog — enough outputs to make Kappa worth revisiting, and enough variety to stress-test the embedding pipeline somewhere TF-IDF might actually struggle. And a lighter-weight version of the human check seems doable: only spot-check the outputs where the judge's hallucination flag rests on a specific number, instead of re-scoring everything by hand.
Wrapping Up
Part 1 showed the pipeline could work. Part 2 was about whether I could trust an LLM to tell me how well it worked. Mostly, yes — Gemini's scores were directionally useful and cheap to generate at scale. But it consistently mistook correct inference for invention, which is exactly the kind of quiet, high-stakes mistake a human is there to catch. For anything a PM is actually going to read and act on, that check isn't optional.
Code, RAG files, and full scoring tables for both models are on GitHub.
Top comments (0)