DEV Community

Kadavath Priyanka
Kadavath Priyanka

Posted on

My CI Agent Kept Forgetting Everything—Here’s How I Fixed Its Memory!

#agents #ai #python #security

Persistent Memory Makes Prompt Injection Worse. Here's How We Guarded Ours

We built a multi-agent competitive intelligence pipeline in CrewAI. You give it a topic and competitors, and agents research the web, analyse what they find, and write a briefing where every claim carries a citation. It worked, but it forgot everything between runs. Week 8 knew nothing about weeks 1 to 7.

Fixing that meant adding persistent memory. It also created a problem we hadn't had before: anything an agent stores can come back in every future run.

Why memory changes the threat

The pipeline reads web pages. If a page contains hidden instructions and an agent stores them as a "finding", that text is now in memory. Every later run retrieves it and feeds it to the Analyst. A one-off injection becomes a permanent one. That's why we treated the write path as seriously as the read path.

The memory layer

The memory is a class we wrote, HindsightStore (memory/hindsight_store.py). It's plain Python, using an append-only events.jsonl and JSON files for profiles, strategies, and predictions. It is our own code, not the Vectorize Hindsight library: https://github.com/vectorize-io/hindsight

Findings are stored as typed events (CompetitorEvent) with a competitor, event type, date, title, description, impact score, confidence, and evidence URLs. Agents reach the store through one CrewAI tool, HindsightStoreTool, with six operations: store_event, get_history, get_profile, search_memory, get_strategy, and get_predictions.

The pipeline has seven agents: Discovery, Research, Memory, Analyst, Strategy Evolution, Prediction, and Writer. The Memory Agent stores every finding, then passes stored history and a profile to the Analyst.

The guards

  • Injection guard. Strips instruction-like patterns from fetched pages before they reach the LLM.
  • Memory injection guard. memory_injection_guard() checks queries headed for the store.
  • Input validation. validate_competitor_name() rejects names containing shell metacharacters or longer than 200 characters.
  • Citation guard. Runs on every briefing so claims can't float free of sources.

Typed events help here too. A schema with fixed event types is a smaller surface than free text, because a stored record has to fit a shape.

Evidence-scaled confidence

Memory also lets us make the output less overconfident. Each write updates a per-competitor profile:

high_impact = [e for e in self._events
               if e.competitor == event.competitor and e.impact_score >= 8.0]
if len(high_impact) >= 3:
    prof.risk_level = RiskLevel.CRITICAL
elif len(high_impact) >= 1:
    prof.risk_level = RiskLevel.HIGH
elif prof.total_events >= 5:
    prof.risk_level = RiskLevel.MEDIUM

prof.confidence_score = min(0.98, 0.3 + (prof.total_events * 0.07))
Enter fullscreen mode Exit fullscreen mode

Confidence starts at 0.3, adds 0.07 per event, and caps at 0.98. The Writer hedges its wording when confidence is low.

Before vs. after

We tested on a fictional competitor, "NeuraCode AI", using the seeded demo data that scripts/demo_memory.py runs with no LLM and no network. These are demo events, not real market data.

Before: one event. The Analyst can only say NeuraCode launched a code review tool.

After: six events across product, hiring (15 ML engineers), pricing ($45/seat/month enterprise tier), M&A ($28M acquisition), a security scanner, and a JetBrains integration. The profile shows 6 events and 72% confidence (0.3 + 6 × 0.07).

That's a controlled demo, not a benchmark, and we haven't measured quality over weeks of live use.

Where the guards are weak

  • Pattern matching only catches patterns we thought of. A new phrasing can slip through, and we haven't red-teamed the guards.
  • Impact scores are LLM-generated with no rule-based floor, so the CRITICAL risk level can shift with model or prompt changes.
  • Predictions are never graded automatically. update_prediction_status() exists but nothing calls it.
  • Strategy output is parsed with regex in crew.py, which is brittle.
  • A fresh store auto-seeds demo data, which can mislead testing.

Takeaways

  1. Persistent memory turns injections from temporary into permanent, so validate before you store.
  2. Constrain what memory can hold with a schema.
  3. Don't call a guard "done" until someone has tried to break it.

Code: https://github.com/24wh1a0510/CI---Copilot

Top comments (0)