Most RAG systems treat support cases like static documents. Embed the whole case, retrieve the closest match, and hope the LLM can extract the right step. This works for knowledge bases but breaks down when you need to retrieve troubleshooting guidance from historical cases that evolved through diagnosis, escalation, and resolution stages.
RAFT (Retrieval-Augmented Framework for Troubleshooting Agents) addresses this by modeling each closed case as a directed chain of timeline entries. Instead of retrieving entire cases, it retrieves at the entry level, surfacing cases whose intermediate states match the active case and returning the parent-case trajectory anchored at the matched state. The result is a retrieval layer that preserves the temporal sequence of troubleshooting steps.
The Problem with Flattening Multi-Stage Cases
Enterprise support cases are workflows, not documents. A typical case progresses through:
- Initial symptom report
- Diagnostic steps and their outcomes
- Escalation to specialized teams
- Configuration changes or patches
- Resolution confirmation
When you embed the entire case as a single vector, you lose the progression. If an active case is stuck at the diagnostic stage, you want to retrieve cases that were stuck at the same point and see what happened next, not cases that eventually resolved through a completely different path.
Existing RAG systems return the most similar closed case, but similarity at the case level does not guarantee similarity at the current stage. You might retrieve a case that shares the same initial symptom but diverged early, leaving the agent with irrelevant guidance.
RAFT Architecture
RAFT introduces three core components:
Timeline Entry Abstraction
Each historical case is decomposed into a sequence of entries. An entry represents a discrete state: a symptom observation, a diagnostic command and its output, an escalation event, or a resolution action. Entries are timestamped and linked in a directed chain.
Entry-Level Retrieval
When an active case queries the retrieval layer, RAFT embeds the current state (the most recent entry) and retrieves the top-k matching entries from historical cases. Each retrieved entry carries a pointer to its parent case and its position in the timeline.
Trajectory Return
Instead of returning just the matched entry, RAFT returns the full trajectory from the matched entry forward. If the active case is at diagnostic step 3 and matches a historical case at the same step, the agent receives steps 3 through resolution from that historical case.
An optional case-level graph links cases through a configurable similarity representation (shared symptoms, affected components, or resolution types). This graph provides a fallback when entry-level retrieval returns no strong matches.
Data Structures and Indexing
RAFT stores each case as a linked list of entries in a vector database. The schema looks like this:
class TimelineEntry:
entry_id: str
case_id: str
timestamp: datetime
entry_type: str # symptom, diagnostic, escalation, resolution
content: str
embedding: List[float]
next_entry_id: Optional[str]
metadata: Dict[str, Any]
class Case:
case_id: str
first_entry_id: str
resolution_status: str
case_embedding: List[float] # optional, for case-level graph
Indexing happens in two passes:
Entry-level index: Each entry is embedded independently and stored in a vector database (FAISS, Pinecone, or Weaviate). The embedding model is fine-tuned on support case language, not general-purpose text.
Case-level graph (optional): Cases are embedded using a pooled representation of all entries or a summary of the resolution path. Edges connect cases with cosine similarity above a threshold or cases that share specific metadata (product version, error code, affected service).
Retrieval queries embed the active case's current state and search the entry-level index. The top-k entries are retrieved, and for each, the system walks the linked list forward to return the trajectory.
State Transition Mapping
The key challenge is mapping an active case's current state to historical entry states. RAFT uses a two-stage retrieval process:
Stage 1: Semantic Match
Embed the active case's most recent entry and retrieve the top-k entries from the entry-level index. This surfaces historical entries that describe similar symptoms, diagnostic outputs, or escalation triggers.
Stage 2: Trajectory Filtering
For each retrieved entry, check if the trajectory from that entry forward is still relevant. If the active case has already tried a diagnostic step that appears later in the retrieved trajectory, filter it out. This prevents the agent from receiving redundant or already-failed guidance.
The filtering step requires maintaining a set of actions already attempted in the active case. This can be a simple hash set of action IDs or a more sophisticated state machine that tracks which branches of the troubleshooting tree have been explored.
Evaluation and Benchmarking
The paper introduces two evaluation datasets:
Synthetic Benchmark (Microsoft Learn Windows Server)
The authors built a synthetic dataset from Windows Server troubleshooting documentation. Each document was decomposed into a multi-stage case with simulated progression. This dataset provides clean ground truth but lacks the noise and variability of real support cases.
Apache Jira Issues
Real Jira issues with human-created duplicate labels serve as a directional validation. If two issues are marked as duplicates, they should share similar intermediate states, not just similar final resolutions. RAFT retrieves at the entry level and checks if the matched entry's parent case is the labeled duplicate.
The primary metric is Case Hit: the percentage of queries where the correct historical case appears in the top-k retrieved trajectories. RAFT improves Case Hit over vanilla RAG and GraphRAG baselines at every stage of case progress, with statistically significant gains.
| Retrieval Method | Case Hit @ k=5 (Early Stage) | Case Hit @ k=5 (Mid Stage) | Case Hit @ k=5 (Late Stage) |
|---|---|---|---|
| Vanilla RAG | 42% | 38% | 35% |
| GraphRAG | 48% | 44% | 40% |
| RAFT | 61% | 57% | 53% |
The drop in Case Hit as cases progress is expected: later stages introduce more variability, and the set of relevant historical cases narrows.
Observability and Failure Modes
RAFT introduces new observability requirements:
Entry Match Quality
Track the cosine similarity distribution of retrieved entries. If most matches fall below 0.7, the entry-level index may be too sparse or the embedding model may not capture support case semantics well.
Trajectory Relevance
Log how often retrieved trajectories are filtered out in Stage 2. High filter rates indicate that entry-level matches are semantically similar but procedurally divergent.
Case Graph Fallback Rate
If the optional case-level graph is enabled, track how often it is invoked. Frequent fallback suggests the entry-level index is missing relevant matches.
Common failure modes:
- Sparse historical data: If few closed cases exist for a specific product or configuration, entry-level retrieval returns weak matches. The case-level graph can help, but it reintroduces the flattening problem.
- Noisy entry boundaries: If entries are chunked poorly (too granular or too coarse), the timeline structure degrades. Automated chunking based on timestamps or action types may require manual tuning.
- Stale embeddings: As product versions change, historical cases become less relevant. Re-embedding old cases with updated models or deprecating cases older than a threshold helps maintain retrieval quality.
Deployment Shape
RAFT fits into a standard RAG pipeline with modifications to the retrieval layer:
- Ingestion: Parse closed cases into timeline entries, embed each entry, and store in a vector database. Build the case-level graph if needed.
- Active Case State Tracking: Maintain a state object for each active case, recording the sequence of entries (symptoms, diagnostics, actions).
- Retrieval: On each agent query, embed the current state, retrieve top-k entries, walk the linked lists to return trajectories, and filter based on already-attempted actions.
- Agent Execution: The agent receives a ranked list of trajectories and selects the next action based on the highest-ranked trajectory that has not been tried.
The retrieval layer can be deployed as a standalone service with a REST or gRPC API. The agent orchestrator calls the retrieval service, receives trajectories, and executes the next action. This separation allows the retrieval layer to be scaled independently and shared across multiple agent types (chat support, automated diagnostics, escalation routing).
Security Boundaries
RAFT introduces new security considerations:
- Case Data Leakage: Historical cases may contain customer-specific information (IP addresses, configuration details, error logs). Entry-level retrieval must enforce access control at the case level, not just the entry level. If a user cannot access a case, none of its entries should be retrievable.
- Trajectory Poisoning: If an attacker can inject malicious entries into historical cases, those entries may be retrieved and executed by agents. Ingestion pipelines must validate entry content and metadata before indexing.
- Embedding Model Bias: If the embedding model is trained on biased or incomplete data, it may systematically miss certain types of cases. Regular audits of retrieval quality across case types and customer segments help detect bias.
Code Example: Entry-Level Retrieval
Here is a simplified retrieval function that queries the entry-level index and returns trajectories:
from typing import List, Dict, Optional
import numpy as np
class RAFTRetriever:
def __init__(self, vector_db, entry_store, k=5):
self.vector_db = vector_db
self.entry_store = entry_store
self.k = k
def retrieve_trajectories(
self,
current_entry_embedding: np.ndarray,
attempted_actions: set
) -> List[Dict]:
# Stage 1: Semantic match
matches = self.vector_db.search(current_entry_embedding, k=self.k)
trajectories = []
for match in matches:
entry_id = match['entry_id']
similarity = match['similarity']
# Walk forward from matched entry
trajectory = self._build_trajectory(entry_id)
# Stage 2: Filter based on attempted actions
if not self._trajectory_overlaps(trajectory, attempted_actions):
trajectories.append({
'case_id': match['case_id'],
'matched_entry_id': entry_id,
'similarity': similarity,
'trajectory': trajectory
})
return sorted(trajectories, key=lambda x: x['similarity'], reverse=True)
def _build_trajectory(self, start_entry_id: str) -> List[Dict]:
trajectory = []
current_id = start_entry_id
while current_id:
entry = self.entry_store.get(current_id)
trajectory.append({
'entry_id': entry.entry_id,
'entry_type': entry.entry_type,
'content': entry.content,
'action_id': entry.metadata.get('action_id')
})
current_id = entry.next_entry_id
return trajectory
def _trajectory_overlaps(
self,
trajectory: List[Dict],
attempted_actions: set
) -> bool:
trajectory_actions = {
e['action_id'] for e in trajectory if e.get('action_id')
}
return bool(trajectory_actions & attempted_actions)
This code assumes a vector database with a search method and an entry store with a get method. The attempted_actions set prevents the agent from receiving trajectories that repeat already-failed steps.
Technical Verdict
Use RAFT when:
- You have a corpus of multi-stage support cases or troubleshooting workflows with clear temporal progression.
- Your agents need to retrieve actionable next steps, not just similar cases.
- You can invest in entry-level chunking and embedding model fine-tuning.
- You need to scale retrieval independently from agent execution.
Avoid RAFT when:
- Your support cases are short and do not evolve through multiple stages.
- You lack sufficient historical cases to populate the entry-level index.
- Your use case requires real-time retrieval with sub-50ms latency (trajectory walking adds overhead).
- You cannot enforce access control at the case level (entry-level retrieval increases the attack surface).
RAFT is a specialized retrieval architecture for troubleshooting agents. It trades simplicity for precision, requiring more complex indexing and state tracking in exchange for better retrieval quality at each stage of case progression. If your agents are drowning in irrelevant historical cases, RAFT provides a clear path to stateful, stage-aware retrieval.
Top comments (0)