DEV Community: Mahendra Gurjar

Why Your AI Agent Keeps Forgetting Things

Mahendra Gurjar — Tue, 10 Feb 2026 04:27:27 +0000

Ever built an AI agent that just... forgets stuff? You tell it something important, and 10 steps later, it's gone.

I spent days debugging this exact problem, and it led me to build MemTrace - a framework that automatically diagnoses why AI agents lose their memories.

The Problem

Imagine you're building a personal assistant agent:

You: "My deadline is Friday"
Agent: "Got it! Your deadline is Friday"

[Agent does 15-20 other things]

You: "When's my deadline?"
Agent: "I don't have that information"

What happened? The agent forgot. But why?

Did it run out of memory space? (Eviction)
Did it overwrite the deadline with something else? (Overwriting)
Did the LLM just hallucinate a wrong answer? (Hallucination)

Without proper diagnosis, you're just guessing. 🎲

The Solution: Event Sourcing for Memory

Here's the key insight: Track every single memory operation as an immutable event.

Instead of just storing data, we log:

✅ Every WRITE (when data is stored)
✅ Every READ (when data is retrieved)
✅ Every UPDATE (when data is overwritten)
✅ Every EVICT (when data is removed due to capacity)

Think of it like a flight recorder for your agent's brain. 🛩️

🏗️ How MemTrace Works

Step 1: Log Everything

Every memory operation creates an event:

MemoryEvent(
    event_type="WRITE",
    step=1,
    key="deadline",
    value="Friday",
    importance=0.9,  # How critical is this data?
    timestamp=1234567890
)

Step 2: Run Automated Tests

Generate 1000+ random scenarios:

Random memory operations (writes and reads)
Different capacity constraints (what if memory is limited?)
Varying importance levels (some data matters more)

Step 3: Auto-Diagnose Failures

For every READ operation, MemTrace automatically:

Finds the original WRITE event
Compares expected vs actual value
If they don't match, traces through the event log to find out why

Example Diagnosis:

❌ FAILURE DETECTED
Key: "deadline"
Expected: "Friday"
Actual: None

🔍 ROOT CAUSE: Memory Evicted
Evidence:
  • Written at step 1 (importance: 0.9)
  • Evicted at step 15 (reason: capacity overflow)
  • Read attempted at step 20

⚠️ CRITICAL FAILURE: High-importance data lost!

📊 What I Learned

After running 1000+ scenarios, here's what the data showed:

Finding #1: Capacity Matters (Obviously)

Low capacity (5 slots):  21% success rate
High capacity (30 slots): 29% success rate

But here's the surprise: Even with 30 slots, 71% of reads still failed!

Important Context: These scenarios are extremely random - agents write and read completely unrelated keys with no semantic connection. Real agents would perform much better because they use context and patterns. The low pass rate reveals the worst-case scenario under chaotic conditions.

Finding #2: Overwrites Are Sneaky

Memory evictions:   ~2200 failures
Memory overwrites:  ~240 failures

Overwrites happen regardless of capacity - they're about key reuse patterns, not memory size.

Finding #3: Critical Failures Are Real

Total memory failures: 2501
Critical failures:     892 (35.7%)

35% of memory failures involved high-importance data. That's your deadlines, user preferences, and key facts - the stuff that actually matters.

🎯 The Architecture (For the Experts)

┌─────────────────┐
│  User Command   │
└────────┬────────┘
         │
         ▼
┌─────────────────────┐
│ StructuredAgent     │ ← Routes to STM or LTM
└────────┬────────────┘
         │
         ▼
┌─────────────────────┐
│ MemoryStore         │ ← Executes operation
│ (STM or LTM)        │
└────────┬────────────┘
         │
         ▼
┌─────────────────────┐
│ MemoryEvent         │ ← Immutable event logged
│ (event_log)         │
└────────┬────────────┘
         │
         ▼
┌─────────────────────┐
│ auto_evaluate_all() │ ← Finds all READ events
└────────┬────────────┘
         │
         ▼
┌─────────────────────┐
│ diagnose_failure()  │ ← Root cause analysis
└─────────────────────┘

Key Design Decisions:

Event Log = Ground Truth: The event log is append-only and never modified. It's the single source of truth for diagnosis.
Importance Tracking: Each event has an importance score (0.0-1.0). Critical failures are flagged when high-importance data (≥0.7) is lost.
Multi-Layer Memory: Separate STM (capacity-limited) and LTM (unlimited) with automatic routing.
Zero Manual Work: auto_evaluate_all() automatically finds every READ event and diagnoses failures without manual test case creation.

🚀 Try It Yourself

git clone -b ltm https://github.com/Mahendra1706/MemTrace.git
cd MemTrace
pip install -e .
python3 run.py

You'll see output like:

============================================================
MEMTRACE RANDOM TESTING - 1000 Scenarios
============================================================

Total Reads: 4993
✅ Passed: 741 (14.8%)
❌ Failed: 4252 (85.2%)

Failure Breakdown:
  • Memory Evicted: 2261
  • Memory Overwritten: 240
  • Invalid Read: 1708

------------------------------------------------------------
CRITICAL FAILURES (High-Importance Data Loss)
------------------------------------------------------------
Total Critical Failures: 892
  • Critical Evictions: 798
  • Critical Overwrites: 94
============================================================

🎓 What This Means for Your Agents

For Beginners:

Test your memory system before deploying
Know why failures happen, don't just guess
Track what matters with importance scores

For Experts:

Event sourcing enables complete audit trails
Statistical testing reveals failure patterns at scale
Importance-based diagnosis separates critical from trivial failures
Multi-layer architecture (STM/LTM) mirrors cognitive science models

🔮 What's Next?

Current Limitations:

Random scenarios: Completely random read/write patterns (worst-case testing)
No semantic understanding: Simple key-value storage, no context awareness
Structured commands: Not integrated with real LLM calls yet
Single-threaded: Sequential execution only

Future Vision: Semantic Memory

The next major upgrade will transform MemTrace from key-value storage to semantic memory:

Current (v1.1):

write("deadline", "Friday")
read("deadline")  # Must match exact key

Future (v2.0 - Semantic Search):

write("My project deadline is Friday", importance=0.9)
read("when is my project due?")  # Semantic match!
# Returns: "Friday" (understands the question relates to deadline)

How it will work:

Embeddings: Convert memories to vector representations
Similarity Search: Find relevant memories based on meaning, not exact keys
Context-Aware Retrieval: Understand relationships between memories
Realistic Behavior: Mimic how human memory actually works

This will make agents perform much better than the current 14.8% pass rate, because they'll retrieve memories based on semantic relevance rather than exact key matches.

Other Planned Features:

Integration with LangChain/AutoGPT
Real-time monitoring dashboard
Advanced eviction policies (LRU, LFU, importance-based)
Consolidation logic (STM → LTM based on importance)
Vector database integration (Pinecone, Weaviate)

📚 References

GitHub: Mahendra1706/MemTrace (see ltm branch for latest)
Inspiration: Event sourcing patterns, MemGPT architecture, cognitive memory models
Related Work: LangChain memory modules, AutoGPT memory systems

💬 Let's Discuss

Have you dealt with memory failures in your AI agents? What strategies worked for you?

It's my first decent project (or that's what I think), so I'd love if you visit the repo or drop a comment!

I Tested 3000+ LLM Agent Memory Operations - Here's What I Found

Mahendra Gurjar — Thu, 29 Jan 2026 06:19:01 +0000

🤔 The Problem

If you've built LLM-based agents, you've probably noticed: they forget things.

A lot.

Your agent remembers the user's name in message 1, forgets it by message 5, and then hallucinates a completely different name by message 10.

But why do agents forget? Is it:

Memory capacity issues?
Information getting overwritten?
The LLM hallucinating?
Something else? Nobody had data. Just anecdotes and frustration. So I built MemTrace to answer this question with actual statistics.

What I Built

-MemTrace** is a testing framework that tracks every single memory operation an agent makes and diagnoses why recalls fail.
Think of it like a "black box recorder" for agent memory.
Core idea:

Track every WRITE, READ, UPDATE, and EVICT operation
Compare what the agent returns vs. what was originally stored
Diagnose failures with evidence from the event log I tested 1000 random scenarios with 3030 memory operations to find patterns.

Key Findings

Finding 1: Agents Forget 60% of the Time

Valid Recall Rate: 39.6%**
That means when an agent tries to recall information it previously stored, it fails 6 out of 10 times.
(This excludes "invalid reads" where the agent tries to read something that was never written - those are test artifacts, not real failures)

Finding 2: Evictions Dominate

Memory Evicted: 46.2% of all failures**
Nearly half of all memory failures happen because the agent ran out of space and had to evict old information.

Breakdown:

Memory Evicted**: 994 failures (46.2%)
Invalid Read**: 815 failures (37.9%)
Memory Overwritten**: 343 failures (15.9%)
LLM Hallucination**: 0 failures (0.0%) *(Note: No hallucinations in this test because I used a deterministic agent. Real LLMs would show hallucinations too.)

Finding 3: Capacity Matters (Proven)

Validated Invariant: Capacity ↑ → Eviction ↓
I tested different memory capacities and found a clear pattern:

Capacity Low (1-5) Evictions 1354 Pass Rate 21.3%
Capacity Medium (10-15) Evictions 1150 Pass Rate 25.6%
Capacity High (20-30) Evictions 994 Pass Rate 29.0%

↑Higher capacity = fewer evictions = better recall.
Seems obvious, but now we have data to prove it.

Finding 4: Overwrites Are Independent

Overwrites stay constant regardless of capacity*
Whether you have capacity of 5 or 30, you get ~340 overwrites per 1000scenarios.
Why? Overwrites depend on how often you reuse the same keys, not how much memory you have.

Event Sourcing

Every memory operation creates an immutable event:

 python
 MemoryEvent(
     event_id="uuid-1234",
     event_type=MemoryEventType.WRITE,
     memory_layer=MemoryLayer.STM,
     step=1,
     timestamp=1706345678.123,
     key="user_name",
     value="Alice",
     metadata={}
 )

The event log becomes the single source of truth.

Automated Diagnosis

When a read fails, MemTrace analyzes the event history:

Scenario: Agent tries to read "deadline" but gets None

Event log shows:

WRITE deadline="Friday" (step 2) EVICT deadline="Friday" (step 5, reason: capacity_overflow) READ deadline=None (step 7)

Diagnosis: "memory_evicted"

Evidence: "Key was written at step 2, evicted at step 5, recall attempted at step 7"

4 Failure Types

Memory Evicted - Removed due to capacity constraints
Memory Overwritten - Updated with different value
Invalid Read - Never written in the first place
LLM Hallucination - Agent returns wrong value despite correct memory

System Invariants (Validated)

After testing 1000 scenarios, these patterns hold:

Capacity ↑ → Eviction ↓
Overwrite ~ independent of capacity
Invalid Read = scenario artifact
Unknown = 0 always (100% failure categorization)

What I Learned

1. Event Sourcing Is Powerful

Having a complete history of every operation makes debugging so much easier.

Instead of guessing why something failed, you can trace back through the exact sequence of events.

2. Capacity Is Critical

If your agent has limited memory, evictions will dominate your failures.

The data shows a clear linear relationship: double the capacity, reduce evictions by ~15%.

3. Overwrites Are Sneaky

Overwrites happen when you reuse keys. They're independent of capacity, which means you can't solve them by just adding more memory.

You need better key management or versioning.

4. Testing Reveals Patterns

Before building MemTrace, I thought hallucinations would be the main issue.
Nope. Evictions are 3x more common (in my tests with deterministic agents).

Real LLMs would show more hallucinations, but capacity is still a huge factor.

Feedback Welcome

This is my first open-source project and I'm still learning!

If you:

Have ideas for improvements
Found a bug
Have questions

Please reach out!

GitHub