Part 3 of a series on auto-grooming Jira backlogs with ML and LLMs. Read Part 1 and Part 2 for the original pipeline and the first Gemini-vs-human evaluation.
Picking Up Where Part 2 Left Off
Part 2 compared two clustering pipelines through two LLMs, judged by Gemini, then checked against my own review. The finding that stuck with me: Gemini kept flagging correct arithmetic as hallucination, specifically on outputs where a model had to compute something rather than just quote it back.
That result raised an obvious follow-up. Was that a Gemini-specific quirk, or something LLM judges do generally? The only way to check is to add a different kind of judge — not another neutral third party, but the two LLMs actually under evaluation, each scoring the other's work. So I had Claude Sonnet 4.5 score all six of GPT-5.5's outputs, and had GPT-5.5 score all six of Claude's, using the same rubric Gemini used.
That gives every output three independent scores instead of two: Gemini, me, and whichever model didn't produce it. Enough to actually answer two things I couldn't in Part 2 — a Kappa score I trust, and a real answer to "which one do I put in front of my manager."
Why Cross-Judging, Not Self-Judging
To be clear about what this setup does and doesn't test: Claude never scores its own output, and neither does GPT-5.5. This isn't a check for a model favoring itself — I don't have that data. It's a check for whether a model is harsher or more lenient toward a competitor's output than a neutral judge and a human are. Different question, still useful.
Weighting the Three Judges
Averaging all three scores equally felt wrong the moment I thought about it. Gemini and the cross-judge are both LLMs, and Part 2 already showed they share a specific blind spot — treating unverified inference as hallucination — on the exact criteria that matter most (Data Synthesis, Age-Aware Reasoning). If I average three raters and two of them share a correlated bias, that bias doesn't cancel out. It gets a two-thirds vote.
So the composite score here is weighted: 50% human, 25% Gemini, 25% cross-judge. Concretely, for every criterion on every output:
composite = 0.5 × Human_score + 0.25 × Gemini_score + 0.25 × Cross-judge_score
Then, since not every criterion applies to every output type, I summed the composite scores across whichever criteria actually applied and divided by the applicable maximum:
% score = sum(composite scores across applicable criteria) / (5 × count of applicable criteria)
That's the number in the winner table below — it answers one question: how good is this output, once the human's judgment is weighted to matter most. And hallucination flags don't get silently averaged away — if I'd already reviewed a specific claim and judged it legitimate (like GPT-5.5's PM report in Part 2), that call stands. If a flag comes up on a claim I hadn't specifically checked, it stays as an open caution rather than getting resolved by majority vote.
Kappa, Now With Enough Data
Part 2 skipped Cohen's Kappa — six outputs per model wasn't enough to trust a coefficient. With three judge pairs now, pooling both models gives 48 rated criterion-output pairs per comparison, which is enough to run a quadratic-weighted Kappa (appropriate here since these are ordinal 1–5 scores, not flat categories, so a 1-point miss shouldn't count the same as a 4-point miss) without over-reading it.
It's worth being precise about what this number is actually answering, because it's a different question than the composite score above. The composite score asks how good is this output. Kappa asks something narrower: when two judges score the same thing, how often do they land in the same place, beyond what you'd expect from chance alone? A judge can agree with another judge constantly while both of them are still wrong — Kappa doesn't know or care which judge is closer to the truth. It only measures consistency.
I ran sklearn.metrics.cohen_kappa_score(weights='quadratic') on each of the three judge pairs, using the pooled 48-pair list each time, and labeled the results with the standard Landis & Koch bins (0.21–0.40 = "fair," 0.41–0.60 = "moderate") since that's the conventional way to read a Kappa value without just quoting a raw decimal:
| Judge pair | Weighted Kappa | Agreement |
|---|---|---|
| Gemini vs. me | 0.365 | Fair |
| Cross-LLM judge vs. me | 0.494 | Moderate |
| Gemini vs. cross-LLM judge | 0.355 | Fair |
I expected the two LLM judges to agree with each other more than either agreed with me. That's not what happened. The cross-family judge — Claude scoring GPT's work, GPT scoring Claude's — landed closer to my own scoring than Gemini did. I don't have a confident explanation for why. Both judges ran on the exact same rubric and prompt, so it isn't a prompting difference — it may just be a trait of these particular models that shows up in how they weigh evidence, rather than anything specific to the cross-judging setup itself. I'd want to see this repeated on another dataset before reading much into it.
So, Which One Wins?
Combining all three judges with the weighting above, Claude Sonnet 4.5 comes out ahead on every artifact type, and within each model, the TF-IDF pipeline beat sentence embeddings every time:
| Artifact | Winning combination | Composite score |
|---|---|---|
| Duplicate List | Claude + TF-IDF | 55.0% |
| Clusters | Claude + TF-IDF | 77.5% |
| PM Insight Report | Claude + TF-IDF | 70.0% |
That's a cleaner result than I expected — I'd assumed the pipelines would split by artifact type the way Part 1 hinted they might. Instead, TF-IDF's simplicity held up across the board, which says something about this being a keyword-dense backlog rather than a general verdict on embeddings versus TF-IDF.
Before treating that as settled, though: the winning PM report is one of two Claude outputs GPT-5.5 flagged during cross-judging, citing specific figures — "8 confirmed duplicate pairs," "39 open TLS/SASL bugs," and "four of six clusters carry high alignment" — as not directly evidenced in the source text. Gemini didn't flag this one, and I hadn't ruled on it myself the way I did for GPT's Part 2 case.
I went back and checked. "8 confirmed duplicate pairs" and "39 open TLS/SASL bugs" both hold up against the raw ticket data — legitimate, not invented. "Four of six clusters carry high alignment" I couldn't confirm either way with the same confidence. So the flag is partially resolved: two of the three claims were correct inference, not hallucination, and one stays an open question. Per the override rule above, only the resolved part gets cleared — the cluster-alignment claim still needs a closer look before this report goes anywhere near a manager.
What I'd Actually Do With This
Present the Claude Sonnet 4.5 + TF-IDF versions of all three artifacts — duplicate list, clusters, PM report — as the primary recommendation. The duplicate-pair and bug-count figures in the PM report checked out against the raw data. The cluster-alignment claim didn't get the same confirmation, so I'd either soften that specific line before sharing the report or verify it against the cluster output directly first.
What This Changes Going Forward
The Kappa numbers reframe how I'll pick a judge next time. A neutral third-party judge sounds like the more rigorous choice on paper, but here it agreed with a human less than a judge with direct experience generating this exact kind of output did. If I'm running this again on a bigger backlog, I'd rather have Claude and GPT cross-check each other's work by default and treat a fully neutral judge as the tiebreaker, not the primary score.
The other lesson: a hallucination flag that nobody has specifically resolved should never get averaged into a composite score as if it were just another data point. It's a to-do item, not a number.
Next Steps
I'd like to see whether the "cross-judge agrees with humans more" pattern holds on a second dataset before trusting it as a general rule rather than a one-off result. And the one remaining open item — the cluster-alignment claim in the winning PM report — is a small check, but worth closing out before this framework's first real recommendation goes to a manager.
Full scoring tables for all three judges, both models, are on Github.
Top comments (0)