DEV Community

Cover image for I built an agent to audit incidents, but it kept forgetting Tuesday by Wednesday.
Karthik S
Karthik S

Posted on

I built an agent to audit incidents, but it kept forgetting Tuesday by Wednesday.

#ai #opensource #backend #javascript

When you build an enterprise AI agent, the honeymoon phase lasts exactly until you push to production.

I built "SentinelOps", an AI agent designed to audit compliance risks and operational incidents for our engineering teams. Initially, it was a massive success. It could ingest a JSON dump of server logs and perfectly diagnose a memory leak, providing step-by-step remediation plans.

But a week later, the exact same memory leak occurred. I asked SentinelOps what to do. It hallucinated three entirely different solutions, completely forgetting the successful remediation plan we executed just days prior.

My brilliant AI had severe amnesia. Here is how I re-architected our agent using Hindsight to give it persistent, semantic memory.

The Problem with Stateless Agents

Large Language Models are inherently stateless. Every time you open a new context window, the model is born yesterday.

Many developers try to solve this by dumping raw conversation logs back into the context window. This is a terrible idea. Not only do you hit token limits incredibly fast, but you also destroy the agent's ability to focus. Feeding an agent 100 pages of raw chat history to help it solve a targeted compliance issue just confuses it.

What I needed was true agent memory—a way for the agent to semantically search its past experiences and only retrieve the exact memories relevant to the current crisis.

Implementing Hindsight

I turned to the Hindsight docs to build a persistent memory layer. Instead of storing raw chat logs, I modified my backend to only store the outcomes and decisions.

Every time SentinelOps successfully diagnosed an issue, it was forced to generate a structured JSON summary. We then piped that summary directly into Hindsight:

// backend/services/memoryService.js
import { HindsightClient } from '@vectorize-io/hindsight-client';

const hindsight = new HindsightClient({ url: process.env.HINDSIGHT_URL });

export async function rememberDecision(interactionId, query, decision) {
  try {
    const memoryDocument = `
      Incident Query: ${query}
      Risk Level: ${decision.riskLevel}
      Remediation: ${decision.recommendedAction}
      Governance: ${decision.governanceSeverity}
    `;

    await hindsight.store({
      id: interactionId,
      content: memoryDocument,
      metadata: {
        domain: decision.domain,
        timestamp: new Date().toISOString()
      }
    });
  } catch (err) {
    console.error("Failed to commit to memory:", err);
  }
}
Enter fullscreen mode Exit fullscreen mode

Now, when a new incident comes in, the first thing the agent does is query its own history:

export async function recallContext(query) {
  const matches = await hindsight.search({ query, topK: 3 });

  if (matches.length > 0) {
    return matches.map(m => m.content).join('\n---\n');
  }
  return null;
}
Enter fullscreen mode Exit fullscreen mode

The CascadeFlow Optimization

Once the agent had memory, the context windows started getting a bit larger, which increased API costs. To mitigate this, I implemented CascadeFlow.

Using the cascadeflow docs, I built a routing engine that checks the complexity of the query before hitting the expensive models. If the query is just a simple policy lookup, it routes to a cheap 8B parameter model. If it's a critical infrastructure failure, it pulls the Hindsight memory and routes to the massive 70B reasoning model.

The Result

The transformation was immediate.

Agent Memory in Action

When a recurring Kubernetes crash happened the following week, SentinelOps didn't guess. It responded with: "Based on a similar incident 4 days ago, this is likely a misconfigured readiness probe in the payment-gateway service. Applying previous remediation plan..."

What I Learned

  1. Don't store raw logs. Storing raw conversation history is garbage-in, garbage-out. Force your agent to summarize its decisions before committing them to memory.
  2. Context is king. Providing an LLM with 3 highly relevant historical examples yields significantly better results than zero-shot prompting it.
  3. Memory requires routing. If you are doing RAG or memory injection, you need an intelligent router like CascadeFlow to manage the compute costs of those larger context windows.

Top comments (0)