Your Agent’s Memory Is a Lie: A Durable, Queryable Memory Layer for Local LLM Agents
Local LLM agents need memory to retain context, but naive implementations can spiral into catastrophic resource exhaustion. We learned this the hard way when our production agent silently consumed 8GB of RAM in 12 hours. The culprit was an unbounded embedding cache that grew indefinitely under real-world workloads. Here is how we built a durable, queryable memory layer that survives production.
The Incident: 8GB OOM in 12 Hours
At 3 AM, the on-call engineer received a Slack alert: the OOM Killer had terminated the agent process. The 8GB RAM instance was fully consumed. Restarts provided temporary relief, but the issue recurred after 6 hours. No CPU spikes or log errors were present. The problem was memory bloat caused by an unbounded Python dictionary caching embeddings.
Each unique LLM input (timestamps, user IDs) generated a new cache entry, resulting in zero cache hits. With 128-dimensional float32 embeddings consuming 512 bytes per entry, 16 million entries equaled 8GB. No alarms or warnings were triggered. The leak was invisible until it was too late.
Debugging: The Nightmare
Initial investigations with memory_profiler showed steady memory growth but no clear source. SQLite database checks confirmed it was only 2GB. Finally, tracemalloc revealed the cache, but only after we added it to monitoring.
The lesson was clear: tests used repeated inputs and short runs, which failed to expose the leak. Reality is far more brutal than any test suite.
The Fix: Bounded LRU + SQLite-Only Storage
Step 1: Kill the Cache (Temporary)
Removing the cache stabilized memory usage at 1.8GB, but embedding generation became a 30ms bottleneck. This was unacceptable for performance-critical applications.
Step 2: Bounded LRU Cache (Hardware-Aware)
We implemented a thread-safe, dual-limited LRU cache to replace the unbounded dictionary. The cache enforces both a maximum number of entries and a hard memory limit.
from collections import OrderedDict
import json
import threading
class BoundedLRUCache:
def __init__(self, max_size=10000, max_bytes=5 * 1024 * 1024): # 5MB hard limit
self.cache = OrderedDict()
self.max_size = max_size
self.max_bytes = max_bytes
self._lock = threading.Lock()
self._current_bytes = 0
def get(self, key):
with self._lock:
if key in self.cache:
self.cache.move_to_end(key)
return self.cache[key]
return None
def put(self, key, value):
value_size = len(json.dumps(value)) # Approximate size
with self._lock:
if key in self.cache:
old_value = self.cache[key]
self._current_bytes -= len(json.dumps(old_value))
self.cache.move_to_end(key)
else:
while (len(self.cache) >= self.max_size or
self._current_bytes + value_size > self.max_bytes):
if not self.cache:
break
_, old_value = self.cache.popitem(last=False)
self._current_bytes -= len(json.dumps(old_value))
self.cache[key] = value
self._current_bytes += value_size
Step 3: SQLite Connection Pooling (Race-Condition Fix)
The original implementation suffered from connection leaks due to async writes, hitting the Linux file descriptor limit of 1024. We fixed this with context managers and a bounded queue.
from queue import Queue
import sqlite3
class DurableMemory:
def __init__(self, db_path, batch_size=100, max_queue=1000):
self.db_path = db_path
self.batch_size = batch_size
self.write_queue = Queue(maxsize=max_queue) # Bounded to prevent OOM
self._stop_event = threading.Event()
self._writer_thread = threading.Thread(target=self._process_writes, daemon=True)
self._writer_thread.start()
def _process_writes(self):
while not self._stop_event.is_set():
batch = []
try:
while (not self.write_queue.empty() and
len(batch) < self.batch_size):
batch.append(self.write_queue.get(timeout=1.0)) # Avoid deadlock
except queue.Empty:
continue
if batch:
with sqlite3.connect(self.db_path, timeout=10) as conn: # Auto-closes
conn.execute("BEGIN TRANSACTION")
try:
for item in batch:
conn.execute(
"INSERT INTO memory_entries (content, embedding, metadata) VALUES (?, ?, ?)",
(item["content"], json.dumps(item["embedding"]), json.dumps(item["metadata"]))
)
conn.commit()
except Exception as e:
conn.rollback()
raise e
Hardware Constraints Validation (8GB RAM)
We validated the solution under strict hardware constraints:
| Config | Peak Memory | Time to OOM | Notes |
|---|---|---|---|
| Original (unbounded) | 8.2GB | 12h | Crashes in production |
| No cache | 1.8GB | Never | Slow (30ms per embedding) |
| Bounded LRU (10k) | 2.1GB | Never | Sub-millisecond cache hit, 5MB max |
| Bounded LRU + Queue | 2.2GB | Never | Thread-safe, no leaks |
Key optimizations included:
-
LRU Cache: Dual limits with
max_sizeandmax_bytesensure strict memory control. Thread safety is guaranteed viathreading.Lock(). -
SQLite: WAL mode (
PRAGMA journal_mode=WAL) enables concurrent reads and writes. A connection timeout of 10 seconds avoids deadlocks. - Queue: A bounded queue with a maximum size of 1000 items prevents OOM if writes stall.
Failure Walkthroughs
1. Cache Eviction Under Load
When 10k unique embeddings flood the cache, the LRU evicts the oldest entries atomically, ensuring no partial evictions. The max_bytes parameter enforces a 5MB hard limit, even if max_size allows more entries.
2. SQLite Connection Storm
With 1000 concurrent writes, the bounded queue blocks new writes if full, applying backpressure. The context manager ensures connections close properly, even if exceptions occur.
3. OOM Killer Avoidance
Under 90 percent system memory usage, the cache and queue bounds guarantee usage remains below 2.2GB. No unbounded structures (e.g., dict, list) are present.
Final Architecture (Hardened)
┌───────────────────────────────────────────────────────┐
│ LLM Agent (Python) │
└───────────────┬───────────────────────┬───────────────┘
│ │
┌───────────────▼───────┐ ┌─────────────▼───────────────┐
│ Bounded LRU Cache │ │ Memory Query Engine │
│ (10k entries / 5MB) │ │ (SQLite + sqlite-vec) │
│ Thread-safe │ │ WAL mode + timeouts │
└───────────────┬───────┘ └─────────────┬───────────────┘
│ │
└───────────────┬───────┘
┌───▼───────┐
│ SQLite │
│ (WAL mode) │
└───────────┘
Open Question: Cache vs. No Cache?
The tradeoff between caching and no caching involves latency, memory usage, and complexity:
| Option | Latency | Memory | Complexity |
|---|---|---|---|
| No Cache | 30ms | 1.8GB | Low |
| Bounded LRU | <1ms | 2.2GB | Medium |
No Cache is simpler and bulletproof in terms of memory, but it introduces a 30ms latency per embedding. Bounded LRU offers faster performance but requires monitoring (e.g., using psutil for max_bytes tuning).
Our choice was Bounded LRU for performance-critical applications, with the addition of max_bytes monitoring in CI/CD pipelines.
How would you balance the tradeoff between memory safety and performance in your own implementations?
Top comments (1)
The failure walkthroughs are the part I'd point people at - an OOM post-mortem that shows the eviction path under load is rarer than it should be.
On your closing question, one thing that changed my answer: for us the size limit and the eviction policy turned out to be separate decisions, and only one of them was written down. We had a memory ceiling configured and no explicit eviction policy, and the default was "refuse writes" rather than "evict". So the cap did exactly what it promised and still took the service down - a full cache stopped accepting new entries instead of dropping old ones. Worth checking which of the two your stack defaults to, because the symptom reads as a bug in the caller.
Two questions about BoundedLRUCache, both genuine:
len(json.dumps(value)) counts characters of a serialised string, and that is not the thing the OOM killer counts. I measured a 768-dimension embedding as a plain Python list just now:
json.dumps length : 15,958 chars -> 20.8 per float
sys.getsizeof sum : 25,368 bytes -> 33.0 per float
ratio : 1.59x
So max_bytes is bounding a number about 1.6x smaller than the footprint it's protecting, and the factor moves with dimension count and dtype. Is that ratio calibrated, or is max_bytes deliberately a proxy with headroom baked in? On an 8 GB box the direction of the error matters.
And the measurement itself allocates: json.dumps on a large value materialises a string about as big as the value before you can decide whether to keep it. Under the storm case in your walkthrough, is the peak you're guarding against the one you measure, or the one you create while measuring?
Last one, and it's the thing I got wrong this week rather than a criticism: you put max_bytes monitoring in CI. Does the CI case actually reach eviction - deliberately overfill, then assert an entry was dropped? I shipped a monitoring rule this week that was green because it could only ever be green, and I only found out because I forced it to be wrong on purpose. A bound that is never crossed in the test is a bound nobody has watched work.