Large language models are only as good as the context you give them. While building an LLM-powered Jira Backlog Analyzer, I learned that simply adding Retrieval-Augmented Generation (RAG) wasn't enough. The real breakthrough came from treating different types of project knowledge as different kinds of organizational memory.
When Good Recommendations Aren't Useful
One of the goals of my Jira Backlog Analyzer was pretty simple: help project managers make sense of hundreds of backlog items. The analyzer groups related Jira tickets, flags potential duplicates, and generates executive summaries covering technical trends and priorities.
The first prototype worked better than I expected. Given nothing but the raw Jira tickets, the LLM could summarize clusters, spot duplicate issues, and even suggest what to do next.
Except a lot of those suggestions were generic to the point of being useless.
Take one cluster of tickets about slow API response times. The model recommended: "Consider optimizing query performance to improve response times." Sounds reasonable. Except a caching layer that had already cut those response times in half had shipped two releases earlier. The recommendation wasn't wrong exactly, it was just out of date.
That kept happening. The model would suggest prioritizing security work without knowing the current program increment was already focused on reliability. The language always sounded confident. The substance often wasn't there. It read well and helped with almost nothing.
So I did what most people do at this point: I added RAG. It helped, but only partway.
The Limitation of a Single Knowledge Base
Most RAG tutorials assume you have one pile of documents. You retrieve the most relevant chunks, stuff them into the prompt, and let the model do the rest. That works fine for a lot of question-answering use cases. Software projects are messier than that. Not all project knowledge is trying to answer the same question.
Once I actually sat down and thought about what a project manager needs to know, it broke into three buckets:
- What's already been delivered?
- How does the system actually work?
- Where is the project heading?
Those are three very different questions, and no single document collection answers all of them well. So instead of one big knowledge base, I split project knowledge into three separate RAG sources. That one decision did more for output quality than any amount of extra context ever did.
Three RAGs, Three Different Roles
I stopped thinking of RAG as "extra documents" at some point and started thinking of each source as a different kind of organizational memory.
Historical Memory: Release Notes
Release notes are the record of what's already shipped: completed features, closed bugs, delivered capabilities. Feed them to the analyzer and it stops recommending work that's already done. It can also reason about how the project evolved release over release, instead of judging every ticket on its own.
Operational Memory: Program Context
This one is trickier to pin down because it's the stuff experienced team members carry around in their heads and almost never write into a ticket. Program goals, system architecture, engineering constraints, stakeholder priorities, the domain jargon nobody bothers to define. Without it, the model can tell you what a ticket says but not why anyone should care. Once I added program context, cluster descriptions got noticeably sharper, because the model could finally connect a ticket to the bigger system it belonged to.
Strategic Memory: FY26 Themes
The last RAG is roadmap information and the strategic themes for the current planning cycle. Release notes look backward; this one looks forward. It ended up mattering most for executive summaries. Instead of just describing what was sitting in the backlog, the analyzer could weigh whether a cluster of tickets actually lined up with where the program was trying to go. Recommendations started reflecting the destination, not just the history.
Under the Hood: Three Indexes, Not One
Each source lives in its own vector index. Release notes, program docs, and roadmap docs are chunked and embedded separately, then pulled through a LangChain retrieval chain scoped to whatever task is running. Keeping them apart instead of merging into one index turns out to matter a lot. Similarity search over a combined index tends to surface whichever source happens to have the most chunks, which usually buries the roadmap themes under a mountain of release-note text. Three smaller indexes, each scoped to a task, just retrieve more precisely than one big one ever will.
Not Every Prompt Needs Every RAG
Here's the lesson that took me a bit too long to learn: more context isn't automatically better context.
At first I fed all three RAG sources into every single prompt. Seemed thorough. In practice it just diluted the useful context with noise.
So I matched sources to task instead:
- Duplicate analysis leans on release history, since knowing what's already been done is what tells you whether two tickets describe the same work.
- Cluster analysis leans on program context and FY26 themes, so the model can actually explain why a group of tickets belongs together.
- Executive summaries pull from all three, weighted toward roadmap themes, to tie technical findings back to where the program is headed.
Cutting prompts down this way improved output quality while making them smaller. Sometimes the better engineering move isn't adding more information. It's adding the right information and leaving the rest out.
| Task | Primary RAG | Background |
|---|---|---|
| Cluster naming & PM recommendations | 2026_theme.md |
project_context.md |
| Duplicate insights | release_notes.md |
project_context.md |
| Executive summary |
2026_theme.md & release_notes.md
|
project_context.md |
An Unexpected Benefit
One thing surprised me during evaluation. I compared outputs across multiple LLMs, scored by both automated LLM judges and human reviewers.
The judges didn't always agree with each other on quality, but they agreed on this: outputs grounded in real project knowledge beat generic software advice, every time.
And the improvement wasn't that summaries got longer or more detailed. They got more specific.
Instead of recommending "improve system performance," the analyzer could tie a recommendation to an actual roadmap priority, a release that had already shipped, or an engineering initiative already underway. That specificity is what project managers actually want, not more words.
A Practical Design Pattern
I don't think of what I built as "adding RAG" anymore. I think of it as organizing institutional knowledge:
- Historical knowledge answers where we've been.
- Operational knowledge explains how the system works.
- Strategic knowledge describes where we're going.
Splitting those responsibilities apart made prompt design simpler, cut out a lot of unnecessary context, and led to recommendations that actually matched what the project needed.
This isn't specific to Jira backlogs, either. Any enterprise LLM application probably has several categories of institutional knowledge floating around, policies, operating procedures, historical records, future plans, and each one might deserve its own retrieval strategy rather than getting dumped into one shared index.
As these systems mature, I suspect the real gains won't come from retrieving more documents. They'll come from organizing what you retrieve more thoughtfully. One RAG is a fine place to start. It just isn't always where you should end up.
RAG files are on GitHub.
Top comments (1)
Multi-RAG makes sense when the sources have different jobs. Backlog analysis often mixes ticket text, comments, history, ownership, and product intent. One retrieval layer can blur those together, while separate retrieval paths can preserve why each fact matters.