DEV Community: K Gann

Adding a Third Kind of Judge: What Cross-LLM Evaluation Changed About the Winner

K Gann — Sat, 25 Jul 2026 01:49:49 +0000

Part 3 of a series on auto-grooming Jira backlogs with ML and LLMs. Read Part 1 and Part 2 for the original pipeline and the first Gemini-vs-human evaluation.

Picking Up Where Part 2 Left Off

Part 2 compared two clustering pipelines through two LLMs, judged by Gemini, then checked against my own review. The finding that stuck with me: Gemini kept flagging correct arithmetic as hallucination, specifically on outputs where a model had to compute something rather than just quote it back.

That result raised an obvious follow-up. Was that a Gemini-specific quirk, or something LLM judges do generally? The only way to check is to add a different kind of judge — not another neutral third party, but the two LLMs actually under evaluation, each scoring the other's work. So I had Claude Sonnet 4.5 score all six of GPT-5.5's outputs, and had GPT-5.5 score all six of Claude's, using the same rubric Gemini used.

That gives every output three independent scores instead of two: Gemini, me, and whichever model didn't produce it. Enough to actually answer two things I couldn't in Part 2 — a Kappa score I trust, and a real answer to "which one do I put in front of my manager."

Why Cross-Judging, Not Self-Judging

To be clear about what this setup does and doesn't test: Claude never scores its own output, and neither does GPT-5.5. This isn't a check for a model favoring itself — I don't have that data. It's a check for whether a model is harsher or more lenient toward a competitor's output than a neutral judge and a human are. Different question, still useful.

Weighting the Three Judges

Averaging all three scores equally felt wrong the moment I thought about it. Gemini and the cross-judge are both LLMs, and Part 2 already showed they share a specific blind spot — treating unverified inference as hallucination — on the exact criteria that matter most (Data Synthesis, Age-Aware Reasoning). If I average three raters and two of them share a correlated bias, that bias doesn't cancel out. It gets a two-thirds vote.

So the composite score here is weighted: 50% human, 25% Gemini, 25% cross-judge. Concretely, for every criterion on every output:

composite = 0.5 × Human_score + 0.25 × Gemini_score + 0.25 × Cross-judge_score

Then, since not every criterion applies to every output type, I summed the composite scores across whichever criteria actually applied and divided by the applicable maximum:

% score = sum(composite scores across applicable criteria) / (5 × count of applicable criteria)

That's the number in the winner table below — it answers one question: how good is this output, once the human's judgment is weighted to matter most. And hallucination flags don't get silently averaged away — if I'd already reviewed a specific claim and judged it legitimate (like GPT-5.5's PM report in Part 2), that call stands. If a flag comes up on a claim I hadn't specifically checked, it stays as an open caution rather than getting resolved by majority vote.

Kappa, Now With Enough Data

Part 2 skipped Cohen's Kappa — six outputs per model wasn't enough to trust a coefficient. With three judge pairs now, pooling both models gives 48 rated criterion-output pairs per comparison, which is enough to run a quadratic-weighted Kappa (appropriate here since these are ordinal 1–5 scores, not flat categories, so a 1-point miss shouldn't count the same as a 4-point miss) without over-reading it.

It's worth being precise about what this number is actually answering, because it's a different question than the composite score above. The composite score asks how good is this output. Kappa asks something narrower: when two judges score the same thing, how often do they land in the same place, beyond what you'd expect from chance alone? A judge can agree with another judge constantly while both of them are still wrong — Kappa doesn't know or care which judge is closer to the truth. It only measures consistency.

I ran sklearn.metrics.cohen_kappa_score(weights='quadratic') on each of the three judge pairs, using the pooled 48-pair list each time, and labeled the results with the standard Landis & Koch bins (0.21–0.40 = "fair," 0.41–0.60 = "moderate") since that's the conventional way to read a Kappa value without just quoting a raw decimal:

Judge pair	Weighted Kappa	Agreement
Gemini vs. me	0.365	Fair
Cross-LLM judge vs. me	0.494	Moderate
Gemini vs. cross-LLM judge	0.355	Fair

I expected the two LLM judges to agree with each other more than either agreed with me. That's not what happened. The cross-family judge — Claude scoring GPT's work, GPT scoring Claude's — landed closer to my own scoring than Gemini did. I don't have a confident explanation for why. Both judges ran on the exact same rubric and prompt, so it isn't a prompting difference — it may just be a trait of these particular models that shows up in how they weigh evidence, rather than anything specific to the cross-judging setup itself. I'd want to see this repeated on another dataset before reading much into it.

So, Which One Wins?

Combining all three judges with the weighting above, Claude Sonnet 4.5 comes out ahead on every artifact type, and within each model, the TF-IDF pipeline beat sentence embeddings every time:

Artifact	Winning combination	Composite score
Duplicate List	Claude + TF-IDF	55.0%
Clusters	Claude + TF-IDF	77.5%
PM Insight Report	Claude + TF-IDF	70.0%

That's a cleaner result than I expected — I'd assumed the pipelines would split by artifact type the way Part 1 hinted they might. Instead, TF-IDF's simplicity held up across the board, which says something about this being a keyword-dense backlog rather than a general verdict on embeddings versus TF-IDF.

Before treating that as settled, though: the winning PM report is one of two Claude outputs GPT-5.5 flagged during cross-judging, citing specific figures — "8 confirmed duplicate pairs," "39 open TLS/SASL bugs," and "four of six clusters carry high alignment" — as not directly evidenced in the source text. Gemini didn't flag this one, and I hadn't ruled on it myself the way I did for GPT's Part 2 case.

I went back and checked. "8 confirmed duplicate pairs" and "39 open TLS/SASL bugs" both hold up against the raw ticket data — legitimate, not invented. "Four of six clusters carry high alignment" I couldn't confirm either way with the same confidence. So the flag is partially resolved: two of the three claims were correct inference, not hallucination, and one stays an open question. Per the override rule above, only the resolved part gets cleared — the cluster-alignment claim still needs a closer look before this report goes anywhere near a manager.

What I'd Actually Do With This

Present the Claude Sonnet 4.5 + TF-IDF versions of all three artifacts — duplicate list, clusters, PM report — as the primary recommendation. The duplicate-pair and bug-count figures in the PM report checked out against the raw data. The cluster-alignment claim didn't get the same confirmation, so I'd either soften that specific line before sharing the report or verify it against the cluster output directly first.

What This Changes Going Forward

The Kappa numbers reframe how I'll pick a judge next time. A neutral third-party judge sounds like the more rigorous choice on paper, but here it agreed with a human less than a judge with direct experience generating this exact kind of output did. If I'm running this again on a bigger backlog, I'd rather have Claude and GPT cross-check each other's work by default and treat a fully neutral judge as the tiebreaker, not the primary score.

The other lesson: a hallucination flag that nobody has specifically resolved should never get averaged into a composite score as if it were just another data point. It's a to-do item, not a number.

Next Steps

I'd like to see whether the "cross-judge agrees with humans more" pattern holds on a second dataset before trusting it as a general rule rather than a one-off result. And the one remaining open item — the cluster-alignment claim in the winning PM report — is a small check, but worth closing out before this framework's first real recommendation goes to a manager.

Full scoring tables for all three judges, both models, are on Github.

Judging the Judges: What Happens When You Ask an LLM to Grade Two Other LLMs

K Gann — Sat, 18 Jul 2026 19:47:01 +0000

Part 2 of a series on auto-grooming Jira backlogs with ML and LLMs. Part 1 article: How I Auto-Groomed 500 Jira Tickets with ML and LLM for the original TF-IDF + Gemini pipeline this builds on.

The Question Part 1 Left Open

Quick recap for anyone who missed Part 1: I pulled 500 real Apache ZooKeeper tickets from Jira's public API, cleaned the text, ran it through TF-IDF and K-Means to get five clusters, then flagged likely duplicates with cosine similarity. That's the part any ML pipeline can do. The interesting part was Phase II — feeding those raw clusters and duplicate pairs into Gemini 2.5 Flash, grounded with three selectively-weighted RAG files (project context, release notes, and a 2026 roadmap theme), so the output came back as named clusters with PM recommendations, age-aware duplicate calls, and an executive summary instead of a spreadsheet of keywords.

But I built that pipeline with exactly one clustering method and one LLM, and never checked it against anything. If Gemini's cluster names sounded sharp, was that because Gemini did a good job, or because there was nothing else in the room to compare it to?

That's the question this part tries to answer. Instead of one pipeline feeding one LLM, I ran two clustering pipelines through two LLMs, producing twelve outputs total, and had a third LLM score all of them against a rubric. Then I scored the same twelve outputs myself, by hand, and compared.

I expected the comparison to tell me something about Claude versus GPT. It ended up telling me something more useful: about where an LLM judge can and can't be trusted.

Why Two Pipelines, Two LLMs

One pipeline gives you no baseline. Adding a second clustering method and a second model lets me vary two things independently:

Clustering approach — sentence embeddings versus TF-IDF. Does semantic embedding actually help on a backlog this keyword-dense, or is TF-IDF's simplicity good enough?
The LLM itself — Claude Sonnet 4.5 versus GPT-5.5, given identical inputs and identical RAG context.

More moving parts than Part 1, but it's the only way to make a claim like "Pipeline A clusters better" or "Claude reasons better about stale tickets" actually mean something, instead of being one anecdote dressed up as a finding.

How It's Wired Together

Five stages, and the diagram below the title maps them out, but here's the short version:

1. Two pipelines, same input. The same 500-ticket CSV runs through Pipeline A (sentence embeddings → K-Means → cosine similarity) and Pipeline B (TF-IDF → K-Means → cosine similarity), independently. Both pipelines picked K the same way — elbow method plus Silhouette Score, rather than reusing Part 1's fixed K=5 outright — so cluster count wasn't a hidden thumb on the scale for either method. Each pipeline spits out its own cluster set and duplicate list.

2. Both LLMs see both pipelines. This is the part that changes from Part 1. Claude Sonnet 4.5 enhances Pipeline A's output, then Pipeline B's. GPT-5.5 does the same — both pipelines, not just a matched one. Each of those four combinations produces the same three artifacts as Part 1: enhanced cluster names and descriptions, an enhanced duplicate list, and a PM Insight Report, grounded in the same three-tier RAG setup (project context, release notes, 2026 theme) and the same age categorization from Part 1.

Two LLMs, two pipelines, three artifacts each — twelve outputs. Having each LLM touch both pipelines matters: it lets me separate "the LLM writes a better PM report" from "the embedding pipeline handed it better raw material," which a matched design (Claude only sees Pipeline A, GPT only sees Pipeline B) would have muddied.

3. A neutral judge scores all twelve. Gemini 3.1 Pro — outside both the Claude and GPT families — scores every output against five criteria (scale 1-5, 5 the best and 1 the worst): Strategic Alignment, PM Recommendation Specificity, Age-Aware Duplicate Reasoning, Cluster Coherence, and Data Synthesis & Quality. Not every criterion applies to every output type (a duplicate list doesn't get scored on cluster coherence, obviously), so each output is scored out of whatever denominator actually applies. The rubric also asks Gemini to flag suspected hallucinations directly, and to score only what's explicitly present rather than inferring intent — a rule that turns out to matter a lot, below.

I'd considered having Claude and GPT also cross-judge each other's outputs, to control for a model favoring its own family. I skipped it here — with only twelve outputs, that's a different (and noisier) experiment than the one I actually wanted to run, which is simpler: does a judge that has no stake in either model agree with a human at all?

4. I reviewed all twelve myself, scoring against the same rubric, before comparing notes with the judge.

5. My review, not the judge's raw score, is what actually determined the final copy for any downstream use of this pipeline.

What the Comparison Actually Showed

The headline: I scored both models' work noticeably higher than Gemini did.

	Gemini avg	My avg	Gap
Claude Sonnet 4.5 outputs	48.9%	70.5%	+21.6 pts
GPT-5.5 outputs	47.8%	63.9%	+16.1 pts

That gap wasn't spread evenly. Breaking it out by criterion:

Criterion	Avg gap (mine − Gemini's)
Data Synthesis & Quality	+1.17
Age-Aware Duplicate Reasoning	+1.13
PM Rec. Specificity	+0.88
Strategic Alignment	+0.58
Cluster Coherence	+0.50

The two biggest gaps — Data Synthesis and Age-Aware Reasoning — are also the two criteria that ask a model to do something with the data rather than just cite it: run arithmetic across ticket counts, or reason about what a ticket's age actually implies. That's not a coincidence.

The clearest example of this showed up in GPT-5.5's PM Insight Report from the embedding pipeline. Gemini flagged it as a hallucination, pointing to specific figures — a claim like "232 issues are Bugs and 147 are Improvements," a "63–121 issues per cluster" range — that it couldn't find written anywhere in the source documents. It scored Data Synthesis a 1.

I gave that same output a 4 on Data Synthesis. The numbers weren't invented — GPT-5.5 had counted and aggregated them from the actual ticket data it was working with. It was correct analysis that happened to produce specific-sounding figures, and because those figures weren't quoted verbatim anywhere in the RAG files, the judge read "not explicitly sourced" as "fabricated."

That's the finding I'd flag to anyone building something similar: an LLM judge, when it can't trace a number back to a quote, seems to default to calling it a hallucination — even when the number is a legitimate computation over data it actually had access to. From the judge's seat, a correct inference and a real hallucination can look identical unless it's willing to redo the math itself.

Worth being honest about the exception, too: on GPT-5.5's TF-IDF PM Report, the pattern flips — Gemini scored Strategic Alignment and Data Synthesis higher than I did. So this isn't "the judge is stingy" across the board. It's specifically the inference-versus-hallucination boundary where the disagreement lives, and it doesn't run one direction consistently enough to just apply a blanket correction factor.

(Full per-criterion scores for both models are in the Github evaluations folder if you want the raw numbers.)

A Note on the Judge Prompt

I won't reproduce the full judge prompt here — it's a fairly long, rigid rubric document, and it's more useful sitting in the repo where you can actually copy and adapt it than pasted into an article. The short version: it told Gemini to act as an impartial judge, gave it explicit 5/3/1 scoring anchors for each of the five criteria, specified which criteria applied to which output type, and included one rule that turned out to matter more than I expected — score only what's explicitly present, don't infer intent. That instruction is almost certainly why the judge treated GPT-5.5's derived figures as unsupported rather than derived: it was doing exactly what it was told, just a little too literally.

Delta, Not Kappa

Cohen's Kappa is the standard tool for this kind of comparison, and I looked at it first. But Kappa needs enough trials to be stable, and with six outputs per model on a 1–5 scale, it would've produced a number I didn't trust. A simple delta — my score minus Gemini's, per criterion — was less elegant but told me exactly where the disagreement was concentrated, which is what actually mattered here.

Where the Human Still Earns Their Keep

The GPT-5.5 hallucination flag above is the whole argument for keeping a person in the loop. It wasn't a vague "I felt more generous" adjustment — it was a specific, checkable disagreement about whether a number was invented or computed. A rubric and a capable judge get you most of the way there, but not all the way; they're not yet reliable at telling "unsupported" from "correct math the judge didn't bother to verify."

What I'd Do Differently Next Time

A few honest takeaways, in no particular order:

The judge is a triage tool, not a verdict. Gemini's scores tracked mine reasonably well most of the time — good enough to flag likely problems fast. Not good enough to be the last word, especially on hallucination calls.

The disagreement wasn't random — it clustered on the two criteria that require inference rather than lookup. That's a fixable design problem: give the judge a way to re-derive a number before flagging it, rather than pattern-matching against the source text alone.

Cross-pipeline coverage was worth the extra setup. Letting each LLM touch both pipelines is the only reason I can say anything about the LLM and the pipeline separately, instead of one tangled result.

What's Next

I want to try giving the judge model an actual calculator or code tool, so it can check a number before calling it fabricated. I'd also like to run this on a bigger, messier backlog — enough outputs to make Kappa worth revisiting, and enough variety to stress-test the embedding pipeline somewhere TF-IDF might actually struggle. And a lighter-weight version of the human check seems doable: only spot-check the outputs where the judge's hallucination flag rests on a specific number, instead of re-scoring everything by hand.

Wrapping Up

Part 1 showed the pipeline could work. Part 2 was about whether I could trust an LLM to tell me how well it worked. Mostly, yes — Gemini's scores were directionally useful and cheap to generate at scale. But it consistently mistook correct inference for invention, which is exactly the kind of quiet, high-stakes mistake a human is there to catch. For anything a PM is actually going to read and act on, that check isn't optional.

Code, RAG files, and full scoring tables for both models are on GitHub.

How I Auto-Groomed 500 Jira Tickets with ML and LLM

K Gann — Wed, 08 Jul 2026 02:42:24 +0000

A practical, end-to-end walkthrough of applying NLP, machine learning, and Gemini to automatically cluster issues, surface duplicates, and generate an executive summary — no manual grooming required.

Why Doesn't Jira Just Do This Already?

If you've inherited a Jira project with hundreds of open tickets, your first instinct is probably: can't the tool just handle this? It's a fair question. Atlassian has poured real investment into AI over the last couple of years, and what they've built — Atlassian Intelligence, and more recently the Rovo platform — genuinely holds up for a lot of tasks.

Atlassian Intelligence can summarize a long comment thread in seconds, and its JQL assistance makes writing complex queries far less painful than it used to be. Rovo goes a step further, with task-specific agents built to automate things like triage and backlog grooming.

What neither tool does, though, is cluster hundreds of tickets into thematic groups or systematically flag duplicates across an entire backlog. Rovo is built around individual ticket interactions: summarizing one issue, routing one request. There's no native way to run a similarity computation across 500 issues at once, or to partition a backlog into clusters a PM can actually act on.

That's the gap this project sits in. What follows is a batch analytical pipeline, which is a different mode of operation entirely from anything Rovo or Atlassian Intelligence currently offers.

What the Research Says

The closest published work I found is a 2025 arXiv paper, "GenAI-Enabled Backlog Grooming in Agile Software Projects," which built a Jira plugin around vector embeddings, cosine similarity, and GPT-4o to propose merges and deletions. It reported 100% precision and a 45% reduction in time-to-completion. This pipeline follows a similar core approach, but diverges in two respects: it uses classical TF-IDF instead of embeddings (no GPU needed), and it adds selective RAG grounding with role-differentiated context files, which the academic study didn't touch.

Earlier research has also shown that ML can predict issue link types across large Jira repositories reasonably well, with duplicate detection being one of the more studied categories. Separately, TF-IDF combined with K-Means has proven effective at filtering large text corpora into focal topics, a pattern that maps cleanly onto backlog clustering.

Where this article tries to add something is the integration layer: stitching these techniques together with a selectively RAG-grounded LLM into one pipeline that ends in a PM-ready executive summary, rather than a set of disconnected outputs.

The Dataset

The sample dataset is 500 publicly available Apache ZooKeeper Jira issues. At a glance, the backlog skews toward core system work: stability, reliability, security, developer experience, and operational scalability all show up repeatedly, and those categories line up well with typical FY26-style strategic themes around hardening and reliability.

The more interesting finding is what's not healthy about the backlog. More than half the tickets are stale or very stale by the age categories defined later in this pipeline, and there's a heavy concentration of bug fixes and major-priority issues sitting in that aged pile. In other words, this isn't a tidy, well-groomed dataset picked to make the pipeline look good — it's a fairly realistic, somewhat neglected backlog, which is exactly the kind of mess this project is meant to help with.

The Pipeline at a Glance

The pipeline runs in two phases:

Phase I: NLP preprocessing → TF-IDF vectorization → K-Means clustering → cosine similarity duplicate detection
Phase II: Selective RAG grounding + Gemini 2.5 Flash → enriched cluster names, age-aware duplicate insights, and an executive summary

Phase I: Clustering and Duplicate Detection

We pull 500 issues from Apache's public Jira REST API, including summary, description, status, priority, labels, and comments. One detail worth flagging early: we store updated as a timezone-aware pandas Timestamp, since it feeds directly into the age categorization in Phase II.

Text preprocessing combines each ticket's summary and description into a single full_text field, then runs it through this cleanup pipeline:

from bs4 import BeautifulSoup
import re
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

def clean_text(text):
    if not isinstance(text, str):
        return []
    text = BeautifulSoup(text, 'html.parser').get_text(separator=' ')
    text = re.sub(r'http\S+|www\S+|https\S+', '', text, flags=re.MULTILINE)
    text = re.sub(r'[^a-zA-Z\s]', '', text)
    text = text.lower()
    tokens = word_tokenize(text)
    stop_words = set(stopwords.words('english'))
    return [w for w in tokens if w not in stop_words]

df_jira_issues['processed_text'] = df_jira_issues['full_text'].apply(clean_text)

From there, standard TF-IDF vectorization produces a (500 × 6,842) matrix, and K-Means with k-means++ initialization sorts issues into 5 clusters. We then compute pairwise cosine similarity across all 500 issues and flag anything above a 0.8 threshold as a potential duplicate.

Why TF-IDF instead of embeddings? ZooKeeper tickets tend to be keyword-dense: error names, component labels, CVE IDs. TF-IDF is fast, interpretable, and doesn't need a GPU. For a backlog with more conversational, prose-heavy tickets, sentence-transformers would probably be the better call.

The cluster sizes came out lopsided: Cluster 3 alone held 286 issues, while Cluster 0 had just 42. That's already a signal on its own — client and server errors dominate this backlog. Cosine similarity turned up 12 potential duplicate pairs. That's where Phase I stops; the raw output here is useful to an engineer but not particularly readable to a PM. Phase II is where that gets fixed.

Phase II: Selective RAG + Gemini

The main design decision in this phase is that the RAG files aren't sent uniformly to every prompt — they're weighted by task. When all three context files carry equal weight, the LLM tends to hedge between historical patterns and forward-looking priorities, and the output gets mushy. Explicit weighting, using prompt instructions like "PRIORITIZE for strategic alignment," produces noticeably sharper results.

Task	Primary RAG	Background
Cluster naming & PM recommendations	`2026_theme.md`	`project_context.md`
Duplicate insights	`release_notes.md`	`project_context.md`
Executive summary	`2026_theme.md` & `release_notes.md`	`project_context.md`

Each file plays a different role. project_context.md is the strategic frame — project goals, workflows, what success looks like — and it shows up in every prompt as background context. release_notes.md holds the historical record of bug fixes and shipped features, and it's the primary driver for duplicate detection and executive summary, since it can tell the LLM whether a reported bug was already resolved in a past release. 2026_theme.md carries the quarterly priorities (for example, FY26Q1: "Core System Hardening") and drives both cluster naming and the executive summary, so the output stays aligned with where the team is actually headed rather than where the backlog happens to be.

Enriching Cluster Names

The cluster prompt asks for a four-field JSON output — name, theme alignment, description, and PM recommendations — grounded primarily in the 2026 themes:

from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_core.prompts import PromptTemplate

llm = ChatGoogleGenerativeAI(model="models/gemini-2.5-flash",
                              temperature=0.3, api_key=GEMINI_API_KEY)

cluster_enhancement_prompt_template = PromptTemplate(
    input_variables=["project_context", "release_notes_context",
                     "theme_2026_context", "cluster_keywords", "cluster_id"],
    template=(
        "You are an expert in Jira issue analysis and project management.\n\n"
        "Project Context (use for overall understanding):\n{project_context}\n\n"
        "Release Notes (use for historical context):\n{release_notes_context}\n\n"
        "2026 Themes (PRIORITIZE for theme alignment and recommendations):\n{theme_2026_context}\n\n"
        "Top keywords for Cluster {cluster_id}: {cluster_keywords}\n\n"
        "Return JSON with keys: name, theme_alignment, description, pm_recommendations.\n"
    )
)

Raw keywords → LLM-enhanced names:

Cluster	Raw Keywords	LLM-Enhanced Name
0	support, auth, apis, admin, prometheus	Admin Server APIs, Authentication, and Prometheus Metrics
1	cve, netty, jetty, vulnerability	Security Vulnerability Fixes and Dependency Upgrades
2	client, session, connection, ssl	ZooKeeper Client Connectivity, Sessions, and Security
3	error, exception, npe, zookeeper	ZooKeeper Client/Server Errors and Stability Issues
4	logging, dependency, core	Core System Dependency and Logging Updates

Cluster 1's name lands on "Fixes and Dependency Upgrades" because 2026_theme.md explicitly calls out dependency remediation as a Core System Hardening priority — the LLM is reading the roadmap here, not just the keywords. Each cluster also comes back with PM-level output: how it connects to FY2026 themes, what the issues represent in practical engineering terms, and what the team should prioritize next quarter.

Age-Aware Duplicate Detection

Before sending duplicate pairs to the LLM, we compute a ticket age category from each issue's updated timestamp:

from datetime import timezone

def categorize_age(updated_date, current_date):
    if pd.isnull(updated_date):
        return "Unknown"
    age_days = (current_date - updated_date).days
    if age_days < 30:   return "Active (< 30 days)"
    elif age_days < 60:  return "Recent (30-60 days)"
    elif age_days < 180: return "Aging (> 60 days)"
    elif age_days < 365: return "Stale (> 180 days)"
    else:                return "Very Stale (> 365 days)"

current_date = pd.Timestamp.now(tz=timezone.utc)

Age gets passed into the duplicate prompt alongside summaries, descriptions, and similarity scores, with release_notes.md set as the primary context. Three real examples from the pipeline output show how much age category alone changes the recommendation, even when the similarity scores are close.

Example 1 — True Duplicate, Stale: ZOOKEEPER-4948 & ZOOKEEPER-4947 (Similarity: 1.0000)

Both tickets are Stale (> 180 days) and describe the exact same bug, almost word-for-word: ZooKeeper's SendThread and EventThread fail to terminate after close() is called during a SASL AuthFailed state. Because these are stale rather than freshly filed, the LLM adds a verification step before recommending any action:

"Both issues are Stale (> 180 days) and relate to a SASL authentication bug introduced in Release 3.8.0. First verify whether a subsequent patch already resolved this. If fixed, close ZOOKEEPER-4947 as a duplicate and close ZOOKEEPER-4948 as Done, referencing the fix. If still open, merge into ZOOKEEPER-4948 and reprioritize as high-severity under FY26Q1 Core System Hardening."

An Active pair with the same similarity score would likely skip that verification caveat entirely. Age is what changes the action here, not the similarity score.

Example 2 — Same Root Cause, Different Components: ZOOKEEPER-4665 & ZOOKEEPER-4664 (Similarity: 0.8624)

Both are Very Stale (> 365 days) and report OWASP failures tied to vulnerable third-party dependencies, but in different components (zooinspector vs. zookeeper-contrib-rest) with entirely different CVE lists. The LLM correctly avoids treating these as exact duplicates:

"These are not identical duplicates but represent the same systemic security problem across two components. Both are Very Stale (> 365 days) — some specific CVEs may have been implicitly resolved through dependency updates in 3.6.3 or 3.8.1. Consolidate under a parent epic 'Address Third-Party Security Vulnerabilities' aligned with FY26Q1 Core System Hardening, and verify current CVE status before prioritizing each sub-task."

The Very Stale category surfaces something a plain similarity check would miss entirely: the specific CVEs involved may not even exist anymore. So the recommendation is to verify first, not to close blindly.

Example 3 — Scope Diverged, Keep Both: ZOOKEEPER-4634 & ZOOKEEPER-4633 (Similarity: 0.9412)

Both are Very Stale (> 365 days) and request authentication support for the same Admin Server APIs, referencing an identical design document. But they diverge on implementation — ZOOKEEPER-4634 targets x509 auth, ZOOKEEPER-4633 targets digest auth. The LLM picks up on the shared origin but treats them as having grown into separate workstreams:

"These are not duplicates — they are two implementation paths for a single feature. Both are Very Stale (> 365 days), and the Admin Server APIs they target were only introduced in Release 3.7.0 (February 2026), likely after these tickets were created. Verify whether either method has already been implemented. If not, consolidate under a parent epic 'Admin Server API Authentication Strategy' and determine whether both authentication types are still required or if the 3.8.0 SASL implementation has superseded them."

This is the case where age and release-note context together catch something neither could catch alone: the tickets were written before the feature they reference even existed, which makes their original requirements possibly obsolete.

Example	Category	What age adds
ZOOKEEPER-4948/4947	True duplicate	Triggers "verify fix before closing" on stale tickets
ZOOKEEPER-4665/4664	Same root cause, different components	Flags possible implicit resolution for very stale CVEs
ZOOKEEPER-4634/4633	Scope diverged	Exposes requirements written before the feature existed

The Output: Executive Summary for PMs

The executive summary prompt leverages RAG by incorporating release_notes.md, 2026_theme.md, and program_context.md. Rather than merely summarizing historical Jira activity, it combines release history, organizational priorities, program context, cluster theme alignments, and duplicate insights to generate a forward-looking 3–5 paragraph executive summary for project managers. The resulting report is saved as executive_summary.md.

It covers a methodology overview, the 5 named clusters with FY2026 theme alignment and PM recommendations, all 12 duplicate pairs with age categories and specific actions, and a closing tied back to quarterly priorities. It's the one artifact meant to bridge engineering analysis and program management without anyone having to open a code cell.

Key Tradeoffs and Lessons Learned

Selective RAG weighting turned out to be the highest-leverage design choice by a wide margin. Moving from uniform context injection to task-specific prioritization made cluster names more strategically forward-looking and duplicate recommendations more historically grounded, without adding much complexity to the pipeline itself.

Ticket age is a cheap addition that punches above its weight. A Very Stale duplicate pair can often be closed with confidence; an Active pair usually needs more digging first. One extra function call adds a dimension that meaningfully shifts the LLM's recommendations.

Cluster imbalance is worth treating as a finding rather than a flaw. 286 issues in Cluster 3 against 42 in Cluster 0 tells you something real about where engineering effort is concentrated. For a production version of this pipeline, running silhouette scoring to find a better k would be a reasonable next step — 5 may simply be too few clusters for a dataset like this one.

The 60-second API delays add up more than expected. With 12 duplicate pairs and a rate-limit delay between calls, Phase II takes roughly 12 minutes end-to-end. Batching or async calls would speed this up considerably at scale.

Next Steps

Auto-triage new tickets: apply the trained K-Means model to incoming issues at creation time for automatic cluster routing.
Threshold tuning dashboard: a Streamlit tool letting PMs adjust the 0.8 similarity threshold and preview duplicates in real time.
Dynamic RAG updates: automate refresh of release_notes.md and 2026_theme.md from Confluence via API, so context stays current without manual upkeep.
Richer embeddings: replace TF-IDF with sentence-transformers for semantically richer clustering on backlogs with more conversational language.

Conclusion

What started as a 500-ticket mess turned into a structured, LLM-annotated backlog report a PM can actually act on, all from a single Jupyter notebook.

The core idea underneath all of it: classical ML handles the heavy lifting (TF-IDF, K-Means, cosine similarity — fast, and no GPU needed), while a selectively RAG-grounded LLM handles the last mile, turning keyword lists into language that's strategically aligned, age-aware, and historically informed enough for both engineers and program managers to use.

The selective RAG weighting is really the detail that makes the whole thing work. Routing the right knowledge source to the right task — release notes for historical duplicate reasoning, 2026 themes for forward-looking cluster strategy — is what makes the output feel genuinely actionable instead of generically descriptive.

Full code is available on GitHub.

References

A clustering approach for topic filtering within systematic literature reviews — NCBI, 2020. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7078380/
Apache ZooKeeper Public Jira Issue Tracker — Apache Software Foundation. https://issues.apache.org/jira/projects/ZOOKEEPER/issues
Atlassian Rovo — Feature overview. https://www.atlassian.com/software/rovo/features
Automated Detection of Typed Links in Issue Trackers — arXiv, 2022. https://arxiv.org/pdf/2206.07182
GenAI-Enabled Backlog Grooming in Agile Software Projects — arXiv, 2025. https://arxiv.org/pdf/2507.10753
Rovo by Atlassian: What the AI can and can't do — eesel AI blog. https://www.eesel.ai/blog/rovo ineering effort is concentrated. For a production pipeline, run silhouette scoring to find a better k — 5 may be too few for this dataset.