My CI Agent Kept Forgetting Everything—Here’s How I Fixed Its Memory!

Kadavath Priyanka — Tue, 29 Sep 2026 18:03:15 +0000

Persistent Memory Makes Prompt Injection Worse. Here's How We Guarded Ours

We built a multi-agent competitive intelligence pipeline in CrewAI. You give it a topic and competitors, and agents research the web, analyse what they find, and write a briefing where every claim carries a citation. It worked, but it forgot everything between runs. Week 8 knew nothing about weeks 1 to 7.

Fixing that meant adding persistent memory. It also created a problem we hadn't had before: anything an agent stores can come back in every future run.

Why memory changes the threat

The pipeline reads web pages. If a page contains hidden instructions and an agent stores them as a "finding", that text is now in memory. Every later run retrieves it and feeds it to the Analyst. A one-off injection becomes a permanent one. That's why we treated the write path as seriously as the read path.

The memory layer

The memory is a class we wrote, HindsightStore (memory/hindsight_store.py). It's plain Python, using an append-only events.jsonl and JSON files for profiles, strategies, and predictions. It is our own code, not the Vectorize Hindsight library: https://github.com/vectorize-io/hindsight

Findings are stored as typed events (CompetitorEvent) with a competitor, event type, date, title, description, impact score, confidence, and evidence URLs. Agents reach the store through one CrewAI tool, HindsightStoreTool, with six operations: store_event, get_history, get_profile, search_memory, get_strategy, and get_predictions.

The pipeline has seven agents: Discovery, Research, Memory, Analyst, Strategy Evolution, Prediction, and Writer. The Memory Agent stores every finding, then passes stored history and a profile to the Analyst.

The guards

Injection guard. Strips instruction-like patterns from fetched pages before they reach the LLM.
Memory injection guard. memory_injection_guard() checks queries headed for the store.
Input validation. validate_competitor_name() rejects names containing shell metacharacters or longer than 200 characters.
Citation guard. Runs on every briefing so claims can't float free of sources.

Typed events help here too. A schema with fixed event types is a smaller surface than free text, because a stored record has to fit a shape.

Evidence-scaled confidence

Memory also lets us make the output less overconfident. Each write updates a per-competitor profile:

high_impact = [e for e in self._events
               if e.competitor == event.competitor and e.impact_score >= 8.0]
if len(high_impact) >= 3:
    prof.risk_level = RiskLevel.CRITICAL
elif len(high_impact) >= 1:
    prof.risk_level = RiskLevel.HIGH
elif prof.total_events >= 5:
    prof.risk_level = RiskLevel.MEDIUM

prof.confidence_score = min(0.98, 0.3 + (prof.total_events * 0.07))

Confidence starts at 0.3, adds 0.07 per event, and caps at 0.98. The Writer hedges its wording when confidence is low.

Before vs. after

We tested on a fictional competitor, "NeuraCode AI", using the seeded demo data that scripts/demo_memory.py runs with no LLM and no network. These are demo events, not real market data.

Before: one event. The Analyst can only say NeuraCode launched a code review tool.

After: six events across product, hiring (15 ML engineers), pricing ($45/seat/month enterprise tier), M&A ($28M acquisition), a security scanner, and a JetBrains integration. The profile shows 6 events and 72% confidence (0.3 + 6 × 0.07).

That's a controlled demo, not a benchmark, and we haven't measured quality over weeks of live use.

Where the guards are weak

Pattern matching only catches patterns we thought of. A new phrasing can slip through, and we haven't red-teamed the guards.
Impact scores are LLM-generated with no rule-based floor, so the CRITICAL risk level can shift with model or prompt changes.
Predictions are never graded automatically. update_prediction_status() exists but nothing calls it.
Strategy output is parsed with regex in crew.py, which is brittle.
A fresh store auto-seeds demo data, which can mislead testing.

Takeaways

Persistent memory turns injections from temporary into permanent, so validate before you store.
Constrain what memory can hold with a schema.
Don't call a guard "done" until someone has tried to break it.

Code: https://github.com/24wh1a0510/CI---Copilot

DEV Community: Kadavath Priyanka