How I Auto-Groomed 500 Jira Tickets with ML and LLM

K Gann — Wed, 08 Jul 2026 02:42:24 +0000

A practical, end-to-end walkthrough of applying NLP, machine learning, and Gemini to automatically cluster issues, surface duplicates, and generate an executive summary — no manual grooming required.

Why Doesn't Jira Just Do This Already?

If you've ever inherited a Jira project with hundreds of open tickets, your first instinct might be: can't the tool handle this? It's a fair question. Atlassian has invested heavily in AI in recent years, and their answer — Atlassian Intelligence and the newer Rovo platform — is genuinely impressive for many tasks.

Atlassian Intelligence's summarization tool can read through a long comment thread and provide a concise summary, and its JQL assistance lowers the barrier to writing complex queries. Rovo goes further: Rovo Agents are task-specific assistants designed to automate workflows like triaging issues and grooming backlogs.

But here's what neither tool actually does: cluster hundreds of tickets into thematic groups, or systematically detect duplicates at scale across an entire backlog. Rovo is optimized for individual ticket interactions — summarizing one issue, routing one request. It has no native capability to run a similarity computation across 500 issues or partition a backlog into coherent clusters a PM can act on.

What we're building here is a batch analytical pipeline — a fundamentally different mode of operation.

What the Research Says

The closest published work is a 2025 arXiv study, "GenAI-Enabled Backlog Grooming in Agile Software Projects", which built a Jira plugin using vector embeddings, cosine similarity, and GPT-4o to propose merges and deletions — achieving 100% precision while reducing time-to-completion by 45%. Our pipeline validates the same core approach but differs in two ways: we use classical TF-IDF (no GPU required) and introduce selective RAG grounding with role-differentiated context files — something the academic study didn't explore.

Earlier work confirmed that ML can effectively predict issue link types across large Jira repositories, with duplicate detection being one of the most studied categories. And TF-IDF + K-Means has been shown to efficiently filter large text corpora into focal topics — a pattern that maps directly onto backlog clustering.

The gap this article fills is the integration layer: combining these techniques with a selectively RAG-grounded LLM into a single end-to-end pipeline that produces a PM-ready executive summary.

The Pipeline at a Glance

Using 500 publicly available Apache ZooKeeper Jira issues as a sample dataset, the pipeline runs in two phases:

Phase I: NLP preprocessing → TF-IDF vectorization → K-Means clustering → cosine similarity duplicate detection
Phase II: Selective RAG grounding + Gemini 2.5 Flash → enriched cluster names, age-aware duplicate insights, and an executive summary

Phase I: Clustering and Duplicate Detection

We pull 500 issues from Apache's public Jira REST API with fields including summary, description, status, priority, labels, and comments. Critically, we store updated as a timezone-aware pandas Timestamp — this feeds the age categorization in Phase II.

Text preprocessing combines each ticket's summary and description into a full_text field, then cleans it through this pipeline:

from bs4 import BeautifulSoup
import re
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

def clean_text(text):
    if not isinstance(text, str):
        return []
    text = BeautifulSoup(text, 'html.parser').get_text(separator=' ')
    text = re.sub(r'http\S+|www\S+|https\S+', '', text, flags=re.MULTILINE)
    text = re.sub(r'[^a-zA-Z\s]', '', text)
    text = text.lower()
    tokens = word_tokenize(text)
    stop_words = set(stopwords.words('english'))
    return [w for w in tokens if w not in stop_words]

df_jira_issues['processed_text'] = df_jira_issues['full_text'].apply(clean_text)

From there, standard TF-IDF vectorization produces a (500 × 6,842) matrix, and K-Means with k-means++ initialization groups issues into 5 clusters. We then compute pairwise cosine similarity across all 500 issues and flag pairs above a 0.8 threshold as potential duplicates.

Why TF-IDF over embeddings? ZooKeeper tickets are keyword-dense — error names, component labels, CVE IDs. TF-IDF is fast, interpretable, and needs no GPU. For backlogs with richer prose, sentence-transformers would be worth exploring.

The cluster distribution was notably uneven: Cluster 3 contained 286 issues while Cluster 0 had just 42 — already a signal that client/server errors dominate this backlog. Cosine similarity surfaced 12 potential duplicate pairs. Phase I ends here; the raw outputs are meaningful to engineers but opaque to PMs. Phase II fixes that.

Phase II: Selective RAG + Gemini

The key design decision in this pipeline is that RAG files are weighted differently by task, not sent uniformly to every prompt. When all three context files carry equal weight, the LLM hedges between historical patterns and future priorities. Explicit weighting — via prompt instructions like "PRIORITIZE for strategic alignment" — produces sharper outputs.

Task	Primary RAG	Background
Cluster naming & PM recommendations	`2026_theme.txt`	`project_context.txt`
Duplicate insights	`release_notes.txt`	`project_context.txt`
Executive summary	`2026_theme.txt`	`project_context.txt`

The three files serve distinct roles. project_context.txt provides the strategic frame — project goals, workflows, definitions of success — and appears in every prompt as background. release_notes.txt carries the historical record of bug fixes and shipped features; it's the primary driver for duplicate detection because it can tell the LLM whether a reported bug was already fixed in a past release. 2026_theme.txt contains quarterly priorities (e.g., FY26Q1: "Core System Hardening") and drives cluster naming and the executive summary, keeping outputs aligned with where the team is going.

Enriching Cluster Names

The cluster prompt requests a four-field JSON output — name, theme alignment, description, and PM recommendations — grounded primarily in the 2026 themes:

from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_core.prompts import PromptTemplate

llm = ChatGoogleGenerativeAI(model="models/gemini-2.5-flash",
                              temperature=0.3, api_key=GEMINI_API_KEY)

cluster_enhancement_prompt_template = PromptTemplate(
    input_variables=["project_context", "release_notes_context",
                     "theme_2026_context", "cluster_keywords", "cluster_id"],
    template=(
        "You are an expert in Jira issue analysis and project management.\n\n"
        "Project Context (use for overall understanding):\n{project_context}\n\n"
        "Release Notes (use for historical context):\n{release_notes_context}\n\n"
        "2026 Themes (PRIORITIZE for theme alignment and recommendations):\n{theme_2026_context}\n\n"
        "Top keywords for Cluster {cluster_id}: {cluster_keywords}\n\n"
        "Return JSON with keys: name, theme_alignment, description, pm_recommendations.\n"
    )
)

Raw keywords → LLM-enhanced names:

Cluster	Raw Keywords	LLM-Enhanced Name
0	support, auth, apis, admin, prometheus	Admin Server APIs, Authentication, and Prometheus Metrics
1	cve, netty, jetty, vulnerability	Security Vulnerability Fixes and Dependency Upgrades
2	client, session, connection, ssl	ZooKeeper Client Connectivity, Sessions, and Security
3	error, exception, npe, zookeeper	ZooKeeper Client/Server Errors and Stability Issues
4	logging, dependency, core	Core System Dependency and Logging Updates

Cluster 1's name anchors to "Fixes and Dependency Upgrades" because 2026_theme.txt explicitly flags dependency remediation as a Core System Hardening priority. The LLM isn't just reading keywords — it's reading the roadmap. Each cluster also returns PM-level output: how it connects to FY2026 themes, what the issues represent in engineering terms, and what the team should prioritize next quarter.

Age-Aware Duplicate Detection

Before sending duplicate pairs to the LLM, we compute a ticket age category from each issue's updated timestamp:

from datetime import timezone

def categorize_age(updated_date, current_date):
    if pd.isnull(updated_date):
        return "Unknown"
    age_days = (current_date - updated_date).days
    if age_days < 30:   return "Active (< 30 days)"
    elif age_days < 60:  return "Recent (30-60 days)"
    elif age_days < 180: return "Aging (> 60 days)"
    elif age_days < 365: return "Stale (> 180 days)"
    else:                return "Very Stale (> 365 days)"

current_date = pd.Timestamp.now(tz=timezone.utc)

Age travels into the duplicate prompt alongside summaries, descriptions, and similarity scores — with release_notes.txt as the primary context. Three real examples from the pipeline output show how age category produces meaningfully different recommendations across three distinct duplicate scenarios.

Example 1 — True Duplicate, Stale: ZOOKEEPER-4948 & ZOOKEEPER-4947 (Similarity: 1.0000)

Both tickets are Stale (> 180 days) and describe the exact same bug word-for-word: ZooKeeper's SendThread and EventThread fail to terminate after close() is called during a SASL AuthFailed state. Because the tickets are stale — not freshly filed — the LLM doesn't just say "close one." It adds a verification step first:

"Both issues are Stale (> 180 days) and relate to a SASL authentication bug introduced in Release 3.8.0. First verify whether a subsequent patch already resolved this. If fixed, close ZOOKEEPER-4947 as a duplicate and close ZOOKEEPER-4948 as Done, referencing the fix. If still open, merge into ZOOKEEPER-4948 and reprioritize as high-severity under FY26Q1 Core System Hardening."

An Active pair with the same similarity score would skip that verification caveat entirely. Age changes the action.

Example 2 — Same Root Cause, Different Components: ZOOKEEPER-4665 & ZOOKEEPER-4664 (Similarity: 0.8624)

Both tickets are Very Stale (> 365 days) and report OWASP failures from vulnerable third-party dependencies — but in different components (zooinspector vs. zookeeper-contrib-rest) with entirely different CVE lists. The LLM correctly avoids calling these exact duplicates:

"These are not identical duplicates but represent the same systemic security problem across two components. Both are Very Stale (> 365 days) — some specific CVEs may have been implicitly resolved through dependency updates in 3.6.3 or 3.8.1. Consolidate under a parent epic 'Address Third-Party Security Vulnerabilities' aligned with FY26Q1 Core System Hardening, and verify current CVE status before prioritizing each sub-task."

The Very Stale age category surfaces a nuance a simple similarity check would miss: the specific CVEs may no longer exist. The recommendation is to verify before acting, not close blindly.

Example 3 — Scope Diverged, Keep Both: ZOOKEEPER-4634 & ZOOKEEPER-4633 (Similarity: 0.9412)

Both tickets are Very Stale (> 365 days) and request authentication support for the same Admin Server APIs, referencing an identical design document. But they diverge on implementation: ZOOKEEPER-4634 targets x509 auth, ZOOKEEPER-4633 targets digest auth. The LLM recognizes they share an origin but have grown into distinct workstreams:

"These are not duplicates — they are two implementation paths for a single feature. Both are Very Stale (> 365 days), and the Admin Server APIs they target were only introduced in Release 3.7.0 (February 2026), likely after these tickets were created. Verify whether either method has already been implemented. If not, consolidate under a parent epic 'Admin Server API Authentication Strategy' and determine whether both authentication types are still required or if the 3.8.0 SASL implementation has superseded them."

Here, age + release note context together expose something critical: the tickets were written before the feature they reference even existed, making their requirements potentially obsolete. Neither the similarity score nor the LLM alone could surface that — it required the release_notes.txt primary context combined with the age signal.

The three examples cover the full spectrum a real backlog surfaces:

Example	Category	What age adds
ZOOKEEPER-4948/4947	True duplicate	Triggers "verify fix before closing" on stale tickets
ZOOKEEPER-4665/4664	Same root cause, different components	Flags possible implicit resolution for very stale CVEs
ZOOKEEPER-4634/4633	Scope diverged	Exposes requirements written before the feature existed

The Output: Executive Summary for PMs

The executive summary prompt intentionally excludes release_notes.txt — the summary is forward-looking, not a historical audit — and synthesizes cluster theme alignments and duplicate insights into 3–5 strategic paragraphs saved as executive_summary.txt.

The file covers: methodology overview, 5 named clusters with FY2026 theme alignment and PM recommendations, all 12 duplicate pairs with age categories and specific actions, and a conclusion tied to quarterly priorities. It's the artifact that bridges engineering analysis and program management, readable without touching any code.

Key Tradeoffs and Lessons Learned

Selective RAG weighting is the highest-leverage design choice. Moving from uniform context injection to task-specific prioritization made cluster names more strategically forward-looking and duplicate recommendations more historically precise — without adding complexity.

Ticket age is a low-cost, high-value addition. A "Very Stale" duplicate pair warrants confident closure; an "Active" pair needs investigation first. One function call adds a dimension that meaningfully changes the LLM's recommendations.

Cluster imbalance is a finding, not a failure. 286 issues in Cluster 3 vs. 42 in Cluster 0 tells you something real about where engineering effort is concentrated. For a production pipeline, run silhouette scoring to find a better k — 5 may be too few for this dataset.

60-second API delays add up. With 12 duplicate pairs and a rate-limit delay between calls, Phase II takes ~12 minutes end-to-end. Batching or async calls would speed this up significantly at scale.

Next Steps

Auto-triage new tickets: Apply the trained K-Means model to incoming issues at creation time for automatic cluster routing.
Threshold tuning dashboard: A Streamlit tool letting PMs adjust the 0.8 similarity threshold and preview duplicates in real time.
Dynamic RAG updates: Automate refresh of release_notes.txt and 2026_theme.txt from Confluence via API, so context stays current without manual maintenance.
Richer embeddings: Replace TF-IDF with sentence-transformers for semantically richer clustering on backlogs with more conversational language.

Conclusion

What started as a 500-ticket mess became a structured, LLM-annotated backlog report — ready for a PM to act on — in a single Jypyter notebook.

The core insight: classical ML handles the heavy lifting (TF-IDF + K-Means + cosine similarity, fast and GPU-free), and a selectively RAG-grounded LLM handles the last mile — turning keyword lists into strategically aligned, age-aware, historically informed language that both engineers and program managers can use.

The selective RAG weighting is the detail that makes the difference. Routing the right knowledge source to the right task — release notes for historical duplicate reasoning, 2026 themes for forward-looking cluster strategy — produces outputs that are genuinely actionable rather than generically descriptive.

Full code is available on GitHub.

References

[1] A clustering approach for topic filtering within systematic literature reviews — NCBI, 2020. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7078380/
[2] Apache ZooKeeper Public Jira Issue Tracker — Apache Software Foundation. https://issues.apache.org/jira/projects/ZOOKEEPER/issues
[3] Atlassian Rovo — Feature overview. https://www.atlassian.com/software/rovo/features
[4] Automated Detection of Typed Links in Issue Trackers — arXiv, 2022. https://arxiv.org/pdf/2206.07182
[5] GenAI-Enabled Backlog Grooming in Agile Software Projects — arXiv, 2025. https://arxiv.org/pdf/2507.10753
[6] Rovo by Atlassian: What the AI can and can't do — eesel AI blog. https://www.eesel.ai/blog/rovo

DEV Community: K Gann