Why standard RAG failed on stack traces until using Hindsight

Krupa G — Sun, 08 Nov 2026 13:09:39 +0000

If you pipe raw production stack traces into a standard vector database, you quickly discover that cosine similarity has no idea what an outage actually means. Two completely unrelated Python tracebacks look 95% identical to an embedding model because they share identical runtime boilerplate, while two incidents sharing the exact same root cause can look completely distinct in text.

When I started building IncidentIQ—an incident response assistant designed to help developers triage recurring failures—I thought standard Retrieval-Augmented Generation (RAG) would be sufficient. The initial architecture was straightforward: take an error log, chunk it, embed it with an off-the-shelf embedding model, store it in a vector database, and retrieve the closest historical post-mortems when an alert fires.

It failed almost immediately in testing.

Naive semantic similarity cannot distinguish between an operational failure mode and the logging framework wrapping it. More importantly, standard vector databases are fundamentally amnesic regarding outcomes: they have no built-in concept of whether a previously suggested fix actually solved the problem or caused a cascading failover.

Here is the technical post-mortem of why traditional RAG fell short on application logs, how we restructured our architecture around Vectorize agent memory using Hindsight, and what the resulting system looks like in production.

The Anatomy of the Problem: Why Stack Traces Break Embeddings

To understand why standard RAG struggles with incident response, consider what a production traceback actually looks like. Whether it comes from Python, Node.js, or Java, roughly 80% of the payload is framework overhead:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 686, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 188, in call_next
    ...

When an embedding model vectorizes this chunk, the dense vector is dominated by Starlette internals, line numbers, and standard library frame wrappers.

If Service A throws a psycopg2.OperationalError because of a saturated database connection pool, and Service B throws a redis.exceptions.ConnectionError because of an invalid DNS record, standard vector similarity will group them closely together simply because both traces share identical FastAPI middleware frames. Conversely, if a developer writes a post-mortem explaining that setting max_pool_size=100 fixed the Postgres bottleneck, that prose shares virtually zero vector space with the raw error log that preceded it.

We ran into three distinct failure modes:

The Boilerplate Trap: Framework wrappers drowned out the actual failure signal.
Context Window Exhaustion: Piping unparsed 15-megabyte container logs directly into an LLM or embedding pipeline created massive latency and blew past context boundaries.
Outcome Blindness: Standard vector search could not distinguish between an approach that worked and an approach that failed. If a past incident log noted "We tried restarting worker pods, but memory bloat persisted," naive retrieval frequently surfaced "Restart worker pods" as a top recommendation.

We needed a system that did two things: first, strip away the syntactic noise before indexing; second, treat incident history not as static documents, but as an evolving memory graph of verified actions and consequences.

System Architecture: Moving from Documents to Memory Banks

Rather than treating logs as generic text documents, we split the pipeline into three decoupled layers: deterministic log parsing, persistent semantic memory, and LLM reasoning.

flowchart TD
    Client["Application Error Log / Alert"] --> Ingest["FastAPI Ingestion Layer"]
    Ingest --> Parser["Deterministic Log Parser & Classifier"]

    subgraph Storage ["Hybrid Storage Architecture"]
        SQLite[("SQLite DB\n(Operational Metadata)")]
        Hindsight[("Hindsight Cloud\n(Semantic Memory Bank)")]
    end

    Parser -->|Extract Features| Hindsight
    Hindsight -->|Recall Past Resolutions & Outages| Orchestrator["Incident Orchestrator"]
    Parser -->|Parsed Context| Orchestrator

    Orchestrator --> LLM["Groq LLM Engine\n(openai/gpt-oss-120b)"]
    LLM --> RecEngine["Prioritized Recommendation Engine"]

    RecEngine --> SQLite
    RecEngine -->|Retain Active Incident| Hindsight
    RecEngine --> ClientResponse["Structured RCA Response"]

    ClientResponse -.-> Feedback["Developer Feedback Loop"]
    Feedback -->|Update State| SQLite
    Feedback -->|Retain Verified Resolution| Hindsight

1. Deterministic Extraction Before Embedding

Before any text reaches an LLM or memory bank, we pass it through a rule-based parser (backend/app/utils/log_parser.py). The parser strips out intermediate framework noise, identifies the language runtime, extracts the terminal exception line, collects file/line references, and generates a bounded excerpt:

def parse_log(raw_log: str) -> ParsedLog:
    clean_log = raw_log.strip()
    timestamp = extract_timestamp(clean_log)
    keywords = extract_keywords(clean_log)
    file_references: List[str] = []

    # Extract Python traceback frames and terminal exception line
    if PYTHON_TRACEBACK_START.search(clean_log):
        for match in PYTHON_FILE_LINE.finditer(clean_log):
            filepath, lineno, func = match.group(1), match.group(2), match.group(3)
            file_references.append(f"{filepath}:{lineno} in {func}")

        exception_matches = list(PYTHON_EXCEPTION_LINE.finditer(clean_log))
        if exception_matches:
            last_match = exception_matches[-1]
            error_type = last_match.group(1).split(".")[-1]
            error_message = last_match.group(2).strip()

    safe_excerpt = create_safe_excerpt(clean_log, max_chars=2500)
    return ParsedLog(
        error_type=error_type,
        error_message=error_message,
        timestamp=timestamp,
        file_references=file_references[:10],
        keywords=keywords,
        log_excerpt=safe_excerpt
    )

By isolating the actual exception type, culprit line numbers, and extracted keywords, we reduce a 500-line crash dump into a dense, clean technical fingerprint.

Integrating Hindsight for Persistent Cross-Incident Memory

Instead of maintaining a homebrewed vector collection or flat index, we integrated Hindsight, following the official Hindsight documentation.

In Hindsight, memories live in isolated scopes called memory banks. Rather than simply calculating vector proximity between documents, Hindsight retains semantic entities, structured metadata, and temporal facts. It exposes two core operations: retain to store structured experiences, and recall to retrieve semantically relevant context based on queries and budgets.

Preserving Memory Across Asynchronous Backends

One practical engineering hurdle we hit was event loop management. hindsight-client's low-level HTTP transport relies on aiohttp.ClientSession. In an asynchronous FastAPI backend where request handlers and test fixtures invoke memory calls across different threads, bare coroutines can trigger RuntimeError: This event loop is already running or Timeout context manager should be used inside a task.

To solve this, we isolated the Hindsight client inside a dedicated background daemon thread running its own persistent event loop:

# backend/app/memory/hindsight_service.py

_hindsight_loop: Optional[asyncio.AbstractEventLoop] = None
_hindsight_thread: Optional[threading.Thread] = None
_loop_lock = threading.Lock()

def _get_hindsight_loop() -> asyncio.AbstractEventLoop:
    global _hindsight_loop, _hindsight_thread
    with _loop_lock:
        if _hindsight_loop is None or not _hindsight_thread.is_alive():
            _hindsight_loop = asyncio.new_event_loop()

            def _runner():
                asyncio.set_event_loop(_hindsight_loop)
                _hindsight_loop.run_forever()

            _hindsight_thread = threading.Thread(
                target=_runner, 
                daemon=True, 
                name="hindsight_persistent_loop"
            )
            _hindsight_thread.start()
        return _hindsight_loop

def _dispatch_hindsight_call(coro_fn, *args, **kwargs):
    """Dispatches async Hindsight calls to the persistent worker loop thread."""
    loop = _get_hindsight_loop()
    future = asyncio.run_coroutine_threadsafe(coro_fn(*args, **kwargs), loop)
    return future.result(timeout=settings.HINDSIGHT_TIMEOUT_SECONDS + 5.0)

This keeps the underlying aiohttp connection pool alive across requests while preventing event loop collisions during sync/async boundaries.

Retaining the Incident Experience

When an incident is first diagnosed, we retain its technical signature into our memory bank using Hindsight's aretain method:

def retain_incident(self, incident_data: Dict[str, Any]) -> Dict[str, Any]:
    content = (
        f"[INCIDENT-RECORD]\n"
        f"Incident ID: {incident_data['id']}\n"
        f"Application: {incident_data['application_name']}\n"
        f"Error Category: {incident_data['error_category']}\n"
        f"Error Type: {incident_data['error_type']}\n"
        f"Error Message: {incident_data['error_message']}\n"
        f"Suspected Root Cause: {incident_data['suspected_root_cause']}\n"
        f"Log Trace Excerpt:\n{incident_data['log_excerpt']}\n"
    )

    metadata = {
        "incident_id": incident_data["id"],
        "application": incident_data["application_name"],
        "error_type": incident_data["error_type"],
        "error_category": incident_data["error_category"]
    }

    tags = ["incident", incident_data["application_name"], incident_data["error_type"]]

    return _dispatch_hindsight_call(
        self._client.aretain,
        bank_id=self.bank_id,
        content=content,
        context="incident-investigation-history",
        metadata=metadata,
        tags=tags
    )

Notice that we don't dump random paragraphs into memory. We format the memory with explicit structural markers ([INCIDENT-RECORD], metadata tags, and bounded excerpts) so that the memory engine indexes semantic entities rather than arbitrary line fragments.

Closing the Loop: Feedback and Provenance

The critical flaw in most AI triage tools is that they fire recommendations into a void. If an LLM suggests "Flush the Redis cache," and flushing the cache causes an avalanche of backend database queries, the system never learns from that failure.

In IncidentIQ, the feedback loop is a first-class citizen. When an engineer finishes investigating an outage, they submit a feedback payload to /api/incidents/{id}/feedback.

We persist the operational metadata in SQLite, but we retain the actual learning experience in Hindsight:

# backend/app/services/feedback_service.py

def submit_feedback(incident_id: str, request: FeedbackSubmitRequest, db: Session):
    incident = db.query(Incident).filter(Incident.id == incident_id).first()
    if not incident:
        raise ValueError(f"Incident '{incident_id}' not found.")

    is_resolved = request.resolution_status == "Resolved"
    incident.status = "resolved" if is_resolved else "not_resolved"

    # Store experience in Hindsight persistent memory
    hindsight_service.retain_feedback(
        incident_id=incident_id,
        resolution_status=request.resolution_status,
        actual_solution=request.actual_solution,
        developer_notes=request.developer_notes,
        incident_context={
            "application_name": incident.application_name,
            "error_type": incident.error_type,
            "suspected_root_cause": incident.suspected_root_cause
        }
    )

Now, when a similar incident occurs weeks later, the recommendation engine (backend/app/services/recommendation_service.py) checks the recalled memories. If a past incident sharing the same technical signature was confirmed Resolved, that exact solution is prioritized at the top of the recommendation list:

for inc in recalled_incidents:
    outcome = (inc.resolution_status or inc.outcome or "").strip()
    actual_fix = (inc.actual_resolution or "").strip()

    if outcome.lower() == "resolved" and actual_fix:
        items.append(RecommendationItem(
            priority=1,
            type="historical_proven",
            title=f"Proven Resolution from Incident {inc.incident_id}",
            action_steps=[
                f"Apply fix confirmed in incident {inc.incident_id}: {actual_fix}",
                "Monitor connection metrics and error rates post-rollout."
            ],
            why_recommended=(
                f"In previous incident {inc.incident_id}, this exact resolution successfully "
                f"solved the {inc.error_type} failure."
            ),
            related_incident_id=inc.incident_id,
            recorded_outcome="Resolved"
        ))

Results in Production: A Concrete Outage Walkthrough

To see how this works in practice, consider what happens when a database connectivity issue hits two separate microservices over time.

Interaction 1: The Initial Outage

Our payment-processor service began rejecting transactions with a MongoDB server selection failure:

[payment-processor] [ERROR] pymongo.errors.ServerSelectionTimeoutError: No replica set members match selector "Primary()", timeout=30000ms.
...
[payment-processor] [WARN] Connection pool exhausted: checked_out=20, max_pool_size=20.

When submitted to the triage pipeline:

The parser identified the category as Database and the exception as ServerSelectionTimeoutError.
Hindsight was queried. Because this was a clean memory bank, zero historical records were returned.
Groq (openai/gpt-oss-120b) analyzed the raw stack trace and connection pool warnings, suggesting that the primary might be undergoing election or the connection pool was undersized.
The incident was saved with ID inc_d162f3d810, and its context was retained in Hindsight.

The on-call engineer diagnosed that transaction volume had spiked during peak hours, exhausting the default client pool limit. They updated the URI parameter to maxPoolSize=100 and submitted feedback:

POST /api/incidents/inc_d162f3d810/feedback
{
  "resolution_status": "Resolved",
  "actual_solution": "Increased MongoDB connection pool maxPoolSize from 20 to 100 in database connection URI.",
  "developer_notes": "Connections were saturated during high checkout concurrency."
}

Hindsight retained this experience, associating maxPoolSize=100 as a verified fix for ServerSelectionTimeoutError under pool saturation.

Interaction 2: The Subsequent Outage

Two weeks later, a different service (orders-service) experienced a failure. The error message had different wording and a distinct stack trace:

[orders-service] [ERROR] MongoNetworkError: connection timed out waiting for available socket.
[orders-service] [FATAL] Database connection pool max connections limit reached. Cannot service order queue.

With standard RAG, the embedding model would search for "MongoNetworkError" and likely surface generic database documentation or unrelated network partition runbooks.

With Hindsight:

The recall query (Database ServerSelectionTimeoutError in orders-service) queried the memory bank.
Hindsight matched the semantic entities (connection pool exhaustion, socket checkout timeout, MongoDB).
It recalled inc_d162f3d810, along with its recorded outcome (Resolved) and actual fix (Increased maxPoolSize from 20 to 100).
The recommendation engine immediately elevated this past fix to Priority 1:

{
  "priority": 1,
  "type": "historical_proven",
  "title": "Proven Resolution from Incident inc_d162f3d810",
  "action_steps": [
    "Apply fix confirmed in incident inc_d162f3d810: Increased MongoDB connection pool maxPoolSize from 20 to 100",
    "Monitor connection metrics and error rates post-rollout."
  ],
  "why_recommended": "Historical incident inc_d162f3d810 was resolved by raising maxPoolSize from 20 to 100, alleviating pool exhaustion.",
  "related_incident_id": "inc_d162f3d810",
  "recorded_outcome": "Resolved"
}

Instead of starting triage from scratch, the engineer immediately had the proven fix from the prior outage, with clear provenance pointing back to the original incident.

Reusable Lessons Learned

If you are building operational assistants or LLM tooling over technical log data, keep these architectural lessons in mind:

1. Structure Logs Before Embedding

Never feed raw, 10,000-token crash logs directly to an embedding model or LLM. Strip out the intermediate framework frames, extract the terminal exception, collect language-specific line references, and bound your excerpt. Clean inputs yield predictable embeddings; raw dumps yield noise.

2. Embeddings Need Semantic Scopes, Not Flat Collections

A flat vector store cannot distinguish between an error signature, an architecture runbook, and a triage conversation. Organizing memories into distinct functional banks with explicit metadata tags (e.g., using Hindsight's memory bank architecture) prevents unrelated operational documents from polluting incident recall.

3. Track Outcomes, Not Just Queries

An incident response tool that doesn't remember whether its previous advice worked is a liability. By explicitly recording whether a fix succeeded or failed, you can programmatically down-rank unhelpful suggestions and promote proven solutions.

4. Separate Operational Records from Semantic Memory

We kept SQLite for relational metadata (pagination, user records, raw logs, audit logs) and used Hindsight strictly for cross-incident semantic memory and recall. Trying to force a relational database to handle semantic memory—or trying to use a memory engine as an operational database—creates unnecessary operational complexity. Use relational stores for bookkeeping; use persistent agent memory for cognition.

The full implementation, schemas, and test suites for IncidentIQ are available on GitHub. For more details on configuring memory banks and client topologies, consult the Hindsight Documentation.

DEV Community: Krupa G