A practical, end-to-end walkthrough of applying NLP, machine learning, and Gemini to automatically cluster issues, surface duplicates, and generate an executive summary — no manual grooming required.
Why Doesn't Jira Just Do This Already?
If you've inherited a Jira project with hundreds of open tickets, your first instinct is probably: can't the tool just handle this? It's a fair question. Atlassian has poured real investment into AI over the last couple of years, and what they've built — Atlassian Intelligence, and more recently the Rovo platform — genuinely holds up for a lot of tasks.
Atlassian Intelligence can summarize a long comment thread in seconds, and its JQL assistance makes writing complex queries far less painful than it used to be. Rovo goes a step further, with task-specific agents built to automate things like triage and backlog grooming.
What neither tool does, though, is cluster hundreds of tickets into thematic groups or systematically flag duplicates across an entire backlog. Rovo is built around individual ticket interactions: summarizing one issue, routing one request. There's no native way to run a similarity computation across 500 issues at once, or to partition a backlog into clusters a PM can actually act on.
That's the gap this project sits in. What follows is a batch analytical pipeline, which is a different mode of operation entirely from anything Rovo or Atlassian Intelligence currently offers.
What the Research Says
The closest published work I found is a 2025 arXiv paper, "GenAI-Enabled Backlog Grooming in Agile Software Projects," which built a Jira plugin around vector embeddings, cosine similarity, and GPT-4o to propose merges and deletions. It reported 100% precision and a 45% reduction in time-to-completion. This pipeline follows a similar core approach, but diverges in two respects: it uses classical TF-IDF instead of embeddings (no GPU needed), and it adds selective RAG grounding with role-differentiated context files, which the academic study didn't touch.
Earlier research has also shown that ML can predict issue link types across large Jira repositories reasonably well, with duplicate detection being one of the more studied categories. Separately, TF-IDF combined with K-Means has proven effective at filtering large text corpora into focal topics, a pattern that maps cleanly onto backlog clustering.
Where this article tries to add something is the integration layer: stitching these techniques together with a selectively RAG-grounded LLM into one pipeline that ends in a PM-ready executive summary, rather than a set of disconnected outputs.
The Dataset
The sample dataset is 500 publicly available Apache ZooKeeper Jira issues. At a glance, the backlog skews toward core system work: stability, reliability, security, developer experience, and operational scalability all show up repeatedly, and those categories line up well with typical FY26-style strategic themes around hardening and reliability.
The more interesting finding is what's not healthy about the backlog. More than half the tickets are stale or very stale by the age categories defined later in this pipeline, and there's a heavy concentration of bug fixes and major-priority issues sitting in that aged pile. In other words, this isn't a tidy, well-groomed dataset picked to make the pipeline look good — it's a fairly realistic, somewhat neglected backlog, which is exactly the kind of mess this project is meant to help with.
The Pipeline at a Glance
The pipeline runs in two phases:
- Phase I: NLP preprocessing → TF-IDF vectorization → K-Means clustering → cosine similarity duplicate detection
- Phase II: Selective RAG grounding + Gemini 2.5 Flash → enriched cluster names, age-aware duplicate insights, and an executive summary
Phase I: Clustering and Duplicate Detection
We pull 500 issues from Apache's public Jira REST API, including summary, description, status, priority, labels, and comments. One detail worth flagging early: we store updated as a timezone-aware pandas Timestamp, since it feeds directly into the age categorization in Phase II.
Text preprocessing combines each ticket's summary and description into a single full_text field, then runs it through this cleanup pipeline:
from bs4 import BeautifulSoup
import re
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
def clean_text(text):
if not isinstance(text, str):
return []
text = BeautifulSoup(text, 'html.parser').get_text(separator=' ')
text = re.sub(r'http\S+|www\S+|https\S+', '', text, flags=re.MULTILINE)
text = re.sub(r'[^a-zA-Z\s]', '', text)
text = text.lower()
tokens = word_tokenize(text)
stop_words = set(stopwords.words('english'))
return [w for w in tokens if w not in stop_words]
df_jira_issues['processed_text'] = df_jira_issues['full_text'].apply(clean_text)
From there, standard TF-IDF vectorization produces a (500 × 6,842) matrix, and K-Means with k-means++ initialization sorts issues into 5 clusters. We then compute pairwise cosine similarity across all 500 issues and flag anything above a 0.8 threshold as a potential duplicate.
Why TF-IDF instead of embeddings? ZooKeeper tickets tend to be keyword-dense: error names, component labels, CVE IDs. TF-IDF is fast, interpretable, and doesn't need a GPU. For a backlog with more conversational, prose-heavy tickets,
sentence-transformerswould probably be the better call.
The cluster sizes came out lopsided: Cluster 3 alone held 286 issues, while Cluster 0 had just 42. That's already a signal on its own — client and server errors dominate this backlog. Cosine similarity turned up 12 potential duplicate pairs. That's where Phase I stops; the raw output here is useful to an engineer but not particularly readable to a PM. Phase II is where that gets fixed.
Phase II: Selective RAG + Gemini
The main design decision in this phase is that the RAG files aren't sent uniformly to every prompt — they're weighted by task. When all three context files carry equal weight, the LLM tends to hedge between historical patterns and forward-looking priorities, and the output gets mushy. Explicit weighting, using prompt instructions like "PRIORITIZE for strategic alignment," produces noticeably sharper results.
| Task | Primary RAG | Background |
|---|---|---|
| Cluster naming & PM recommendations | 2026_theme.md |
project_context.md |
| Duplicate insights | release_notes.md |
project_context.md |
| Executive summary |
2026_theme.md & release_notes.md
|
project_context.md |
Each file plays a different role. project_context.md is the strategic frame — project goals, workflows, what success looks like — and it shows up in every prompt as background context. release_notes.md holds the historical record of bug fixes and shipped features, and it's the primary driver for duplicate detection and executive summary, since it can tell the LLM whether a reported bug was already resolved in a past release. 2026_theme.md carries the quarterly priorities (for example, FY26Q1: "Core System Hardening") and drives both cluster naming and the executive summary, so the output stays aligned with where the team is actually headed rather than where the backlog happens to be.
Enriching Cluster Names
The cluster prompt asks for a four-field JSON output — name, theme alignment, description, and PM recommendations — grounded primarily in the 2026 themes:
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_core.prompts import PromptTemplate
llm = ChatGoogleGenerativeAI(model="models/gemini-2.5-flash",
temperature=0.3, api_key=GEMINI_API_KEY)
cluster_enhancement_prompt_template = PromptTemplate(
input_variables=["project_context", "release_notes_context",
"theme_2026_context", "cluster_keywords", "cluster_id"],
template=(
"You are an expert in Jira issue analysis and project management.\n\n"
"Project Context (use for overall understanding):\n{project_context}\n\n"
"Release Notes (use for historical context):\n{release_notes_context}\n\n"
"2026 Themes (PRIORITIZE for theme alignment and recommendations):\n{theme_2026_context}\n\n"
"Top keywords for Cluster {cluster_id}: {cluster_keywords}\n\n"
"Return JSON with keys: name, theme_alignment, description, pm_recommendations.\n"
)
)
Raw keywords → LLM-enhanced names:
| Cluster | Raw Keywords | LLM-Enhanced Name |
|---|---|---|
| 0 | support, auth, apis, admin, prometheus | Admin Server APIs, Authentication, and Prometheus Metrics |
| 1 | cve, netty, jetty, vulnerability | Security Vulnerability Fixes and Dependency Upgrades |
| 2 | client, session, connection, ssl | ZooKeeper Client Connectivity, Sessions, and Security |
| 3 | error, exception, npe, zookeeper | ZooKeeper Client/Server Errors and Stability Issues |
| 4 | logging, dependency, core | Core System Dependency and Logging Updates |
Cluster 1's name lands on "Fixes and Dependency Upgrades" because 2026_theme.md explicitly calls out dependency remediation as a Core System Hardening priority — the LLM is reading the roadmap here, not just the keywords. Each cluster also comes back with PM-level output: how it connects to FY2026 themes, what the issues represent in practical engineering terms, and what the team should prioritize next quarter.
Age-Aware Duplicate Detection
Before sending duplicate pairs to the LLM, we compute a ticket age category from each issue's updated timestamp:
from datetime import timezone
def categorize_age(updated_date, current_date):
if pd.isnull(updated_date):
return "Unknown"
age_days = (current_date - updated_date).days
if age_days < 30: return "Active (< 30 days)"
elif age_days < 60: return "Recent (30-60 days)"
elif age_days < 180: return "Aging (> 60 days)"
elif age_days < 365: return "Stale (> 180 days)"
else: return "Very Stale (> 365 days)"
current_date = pd.Timestamp.now(tz=timezone.utc)
Age gets passed into the duplicate prompt alongside summaries, descriptions, and similarity scores, with release_notes.md set as the primary context. Three real examples from the pipeline output show how much age category alone changes the recommendation, even when the similarity scores are close.
Example 1 — True Duplicate, Stale: ZOOKEEPER-4948 & ZOOKEEPER-4947 (Similarity: 1.0000)
Both tickets are Stale (> 180 days) and describe the exact same bug, almost word-for-word: ZooKeeper's SendThread and EventThread fail to terminate after close() is called during a SASL AuthFailed state. Because these are stale rather than freshly filed, the LLM adds a verification step before recommending any action:
"Both issues are Stale (> 180 days) and relate to a SASL authentication bug introduced in Release 3.8.0. First verify whether a subsequent patch already resolved this. If fixed, close ZOOKEEPER-4947 as a duplicate and close ZOOKEEPER-4948 as Done, referencing the fix. If still open, merge into ZOOKEEPER-4948 and reprioritize as high-severity under FY26Q1 Core System Hardening."
An Active pair with the same similarity score would likely skip that verification caveat entirely. Age is what changes the action here, not the similarity score.
Example 2 — Same Root Cause, Different Components: ZOOKEEPER-4665 & ZOOKEEPER-4664 (Similarity: 0.8624)
Both are Very Stale (> 365 days) and report OWASP failures tied to vulnerable third-party dependencies, but in different components (zooinspector vs. zookeeper-contrib-rest) with entirely different CVE lists. The LLM correctly avoids treating these as exact duplicates:
"These are not identical duplicates but represent the same systemic security problem across two components. Both are Very Stale (> 365 days) — some specific CVEs may have been implicitly resolved through dependency updates in 3.6.3 or 3.8.1. Consolidate under a parent epic 'Address Third-Party Security Vulnerabilities' aligned with FY26Q1 Core System Hardening, and verify current CVE status before prioritizing each sub-task."
The Very Stale category surfaces something a plain similarity check would miss entirely: the specific CVEs involved may not even exist anymore. So the recommendation is to verify first, not to close blindly.
Example 3 — Scope Diverged, Keep Both: ZOOKEEPER-4634 & ZOOKEEPER-4633 (Similarity: 0.9412)
Both are Very Stale (> 365 days) and request authentication support for the same Admin Server APIs, referencing an identical design document. But they diverge on implementation — ZOOKEEPER-4634 targets x509 auth, ZOOKEEPER-4633 targets digest auth. The LLM picks up on the shared origin but treats them as having grown into separate workstreams:
"These are not duplicates — they are two implementation paths for a single feature. Both are Very Stale (> 365 days), and the Admin Server APIs they target were only introduced in Release 3.7.0 (February 2026), likely after these tickets were created. Verify whether either method has already been implemented. If not, consolidate under a parent epic 'Admin Server API Authentication Strategy' and determine whether both authentication types are still required or if the 3.8.0 SASL implementation has superseded them."
This is the case where age and release-note context together catch something neither could catch alone: the tickets were written before the feature they reference even existed, which makes their original requirements possibly obsolete.
| Example | Category | What age adds |
|---|---|---|
| ZOOKEEPER-4948/4947 | True duplicate | Triggers "verify fix before closing" on stale tickets |
| ZOOKEEPER-4665/4664 | Same root cause, different components | Flags possible implicit resolution for very stale CVEs |
| ZOOKEEPER-4634/4633 | Scope diverged | Exposes requirements written before the feature existed |
The Output: Executive Summary for PMs
The executive summary prompt leverages RAG by incorporating release_notes.md, 2026_theme.md, and program_context.md. Rather than merely summarizing historical Jira activity, it combines release history, organizational priorities, program context, cluster theme alignments, and duplicate insights to generate a forward-looking 3–5 paragraph executive summary for project managers. The resulting report is saved as executive_summary.md.
It covers a methodology overview, the 5 named clusters with FY2026 theme alignment and PM recommendations, all 12 duplicate pairs with age categories and specific actions, and a closing tied back to quarterly priorities. It's the one artifact meant to bridge engineering analysis and program management without anyone having to open a code cell.
Key Tradeoffs and Lessons Learned
Selective RAG weighting turned out to be the highest-leverage design choice by a wide margin. Moving from uniform context injection to task-specific prioritization made cluster names more strategically forward-looking and duplicate recommendations more historically grounded, without adding much complexity to the pipeline itself.
Ticket age is a cheap addition that punches above its weight. A Very Stale duplicate pair can often be closed with confidence; an Active pair usually needs more digging first. One extra function call adds a dimension that meaningfully shifts the LLM's recommendations.
Cluster imbalance is worth treating as a finding rather than a flaw. 286 issues in Cluster 3 against 42 in Cluster 0 tells you something real about where engineering effort is concentrated. For a production version of this pipeline, running silhouette scoring to find a better k would be a reasonable next step — 5 may simply be too few clusters for a dataset like this one.
The 60-second API delays add up more than expected. With 12 duplicate pairs and a rate-limit delay between calls, Phase II takes roughly 12 minutes end-to-end. Batching or async calls would speed this up considerably at scale.
Next Steps
- Auto-triage new tickets: apply the trained K-Means model to incoming issues at creation time for automatic cluster routing.
- Threshold tuning dashboard: a Streamlit tool letting PMs adjust the 0.8 similarity threshold and preview duplicates in real time.
-
Dynamic RAG updates: automate refresh of
release_notes.mdand2026_theme.mdfrom Confluence via API, so context stays current without manual upkeep. -
Richer embeddings: replace TF-IDF with
sentence-transformersfor semantically richer clustering on backlogs with more conversational language.
Conclusion
What started as a 500-ticket mess turned into a structured, LLM-annotated backlog report a PM can actually act on, all from a single Jupyter notebook.
The core idea underneath all of it: classical ML handles the heavy lifting (TF-IDF, K-Means, cosine similarity — fast, and no GPU needed), while a selectively RAG-grounded LLM handles the last mile, turning keyword lists into language that's strategically aligned, age-aware, and historically informed enough for both engineers and program managers to use.
The selective RAG weighting is really the detail that makes the whole thing work. Routing the right knowledge source to the right task — release notes for historical duplicate reasoning, 2026 themes for forward-looking cluster strategy — is what makes the output feel genuinely actionable instead of generically descriptive.
Full code is available on GitHub.
References
- A clustering approach for topic filtering within systematic literature reviews — NCBI, 2020. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7078380/
- Apache ZooKeeper Public Jira Issue Tracker — Apache Software Foundation. https://issues.apache.org/jira/projects/ZOOKEEPER/issues
- Atlassian Rovo — Feature overview. https://www.atlassian.com/software/rovo/features
- Automated Detection of Typed Links in Issue Trackers — arXiv, 2022. https://arxiv.org/pdf/2206.07182
- GenAI-Enabled Backlog Grooming in Agile Software Projects — arXiv, 2025. https://arxiv.org/pdf/2507.10753
-
Rovo by Atlassian: What the AI can and can't do — eesel AI blog. https://www.eesel.ai/blog/rovo
ineering effort is concentrated. For a production pipeline, run silhouette scoring to find a better
k— 5 may be too few for this dataset.
Top comments (0)