The Goldfish in the Machine (AI Memory & Local Fixes)

CV PRAJWAL GUPTHA — Mon, 26 Oct 2026 09:44:14 +0000

Abstract: Large Language Models suffer from an architectural condition colloquially known as "goldfish memory"—a phenomenon mathematically formalized as the "Lost in the Middle" (LiM) attention basin. Despite context windows expanding to millions of tokens, physical capacity does not equal functional working memory. This report explores the mathematical causes of context window degradation, contrasts parametric and working memory, and provides an offline-first engineering architecture using Python, SQLite (FTS5 + sqlite-vec), Reciprocal Rank Fusion (RRF), and boundary-weighted prompt structuring to give AI models persistent, private, long-term memory.

Act 1: The Problem — Goldfish Memory & The Attention Basin

Modern Large Language Models (LLMs) are marketed on the size of their context windows, claiming capacities from 128K to over 2M tokens. However, backend engineers quickly observe a frustrating disparity between nominal token capacity and effective recall: as prompt length increases, the model's ability to extract specific details degrades significantly.

1. Context Window Exhaustion vs. Effective Working Memory

Physical token capacity measures how many tokens can be ingested into a model's forward pass before running out of Memory (VRAM) or exceeding positional index boundaries. Functional working memory, by contrast, describes the model's ability to attend to, weigh, and synthesize information distributed across those tokens.

[System Instruction] --> [Middle Context: Neural Dead Zone] --> [User Query]
     (High Recall)               (Severe Amnesia)             (High Recall)

Auto-regressive generation requires calculating scaled dot-product attention across all preceding key-value pairs:

Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d _{k}}) V

To avoid recomputing keys and values at every generation step, inference engines maintain a Key-Value (KV) cache. The computational and VRAM footprint of the KV cache scales quadratically O(N²) with sequence length N. As sequence length expands, memory thrashing and attention dilution reduce the effective signal-to-noise ratio of intermediate tokens.

2. The "Lost in the Middle" Phenomenon

Information retrieval accuracy in long contexts follows a characteristic U-shaped curve. Known as the "Lost in the Middle" (LiM) phenomenon or Attention Basin, LLMs consistently display high recall at the beginning and end of a context window, but suffer severe retrieval degradation in the center.

Recall Accuracy (%)
 100% | \                                                     /
      |  \                                                   /
  50% |   \_________________________________________________/
      |                   Middle Context
   0% +---------------------------------------------------------+
      0 (Primacy)              Relative Token Position        1 (Recency)

Primacy Bias: High retrieval fidelity for information placed at the absolute start of the prompt.
Recency Bias: High retrieval fidelity for information placed at the immediate end of the prompt, adjacent to the final query.
Factorial Dead Zone: A central region where retrieval accuracy drops sharply, often ignoring explicit facts, code snippets, or instructions.

3. Mathematical Foundations of Position Bias

Recent theoretical work proves that this U-shaped attention curve is an inherent geometric property of causal decoders with residual connections, present at network initialization before any fine-tuning or positional encodings take effect.

Layer L   [Token 0] ---> [Token 1] ---> ... ---> [Token N-1] ---> [Token N]
             |              |                        |               |
Residual     +--------------+------------------------+---------------+--> Output
Shortcut                   (Direct Skip Connection Anchor)

Modeling multi-layer causal attention through continuous matrix operations reveals three distinct influence zones across sequence depth H:

The Primacy Tail O(log N): In autoregressive causal attention, token 0 is visible to every subsequent token. Causal masking constructs a directed acyclic graph where early tokens sit at the root of the computation tree. This forces a logarithmic accumulation of gradient influence and creates structural "attention sinks" at sequence-start boundaries.
The Recency Delta O(1): Residual skip connections allow information from the final token N to bypass intermediate transformer blocks and reach the output layer directly. This creates an unattenuated linear shortcut—an isolated O(1) recency anchor that guarantees high attention weight for recent inputs.
The Factorial Dead Zone O(1 / (H-1)!): Middle tokens possess neither root causal positioning nor direct residual shortcuts to the output. As sequence length N and layer depth H increase, attention weights across middle positions suffer repeated fractional Softmax dilution:

ρ_{H} (x) \propto \frac{( - ln x ) ^{H - 1}}{( H - 1 )!}

For deeper networks (e.g., H = 32 or H = 80), the factorial denominator (H - 1)! suppresses middle-context influence density to near zero, creating a mathematically hostile environment for information retrieval.

Act 2: The Two Types of AI Memory

To engineer reliable long-term conversational memory, developers must distinguish between the two primary memory mechanisms in transformer architectures.

PARAMETRIC MEMORY	WORKING MEMORY
Permanent neural weights	Volatile context window
Fixed at training time	Cleared after each request
Expensive to update (fine-tune)	Subject to quadratic KV growth
Static global knowledge	Subject to mid-context dilution

1. Parametric Memory

Parametric memory consists of the static floating-point weights optimized during pre-training and fine-tuning.

Strengths: Invariant to prompt length; stores vast world knowledge, syntax, and reasoning patterns.
Limitations: Highly expensive to update; prone to hallucinations when retrieving exact personal details or specific code symbols; cannot adapt dynamically during a live chat session.

2. Working Memory

Working memory corresponds to the active context window passed during inference.

Strengths: Dynamically customizable per request; allows immediate instruction following and in-context learning.
Limitations: Volatile and ephemeral; subject to quadratic compute scaling and mid-context attention decay.

Relying entirely on expanding working memory leads to latency spikes, high inference costs, and neural amnesia. The solution is to introduce a third memory layer: an External Hippocampus.

Act 3: Giving AI a Hippocampus — The Local Offline Fix

In human biology, the hippocampus consolidates short-term experiences into structured, long-term episodic memory, recalling relevant context when prompted. In software engineering, developers replicate this behavior by pairing local LLMs with an offline, zero-dependency hybrid storage engine.

+-------------------------------------------------------------------------+
|                        HYBRID RETRIEVAL PIPELINE                        |
|                                                                         |
|   User Query (Q)                                                        |
|        |                                                                |
|        +-------------------------+--------------------------+           |
|        |                         |                          |           |
|        v                         v                          v           |
|  [FTS5 Lexical Search]   [sqlite-vec Vector]     [Active Summary St]    |
|  (BM25 Keyword Match)    (Cosine Similarity)     (O(1) Bounded State)   |
|        |                         |                          |           |
|        +------------+------------+                          |           |
|                     |                                       |           |
|                     v                                       |           |
|          [Reciprocal Rank Fusion]                           |           |
|          Score S_RRF = ∑ 1/(60+r)                           |           |
|                     |                                       |           |
|                     v                                       |           |
|          [Top-K Ranked Chunks]                              |           |
|                     |                                       v           |
|                     +-------------------+-------------------+           |
|                                         |                               |
|                                         v                               |
|               [Boundary-Weighted Prompt Reconstruction]                 |
|               - Primacy Zone: System Instructions + Top RRF             |
|               - Dead Zone:    [EMPTY / EXCLUDED]                        |
|               - Recency Zone:   Active Summary + User Query             |
+-------------------------------------------------------------------------+

1. Local Offline Storage: SQLite FTS5 + sqlite-vec

Rather than deploying complex, cloud-hosted vector databases, a lightweight, terminal-ready local solution can be constructed using Python and a single-file SQLite database equipped with two native extensions:

SQLite FTS5 (Lexical Search): Uses the BM25 algorithm to index text tokens. Excels at retrieving exact function names (handleSubmit), specific variable identifiers, error codes (ERR_CONNECTION_REFUSED), and domain acronyms.
sqlite-vec (Semantic Search): A C-based SQLite extension implementing K-Nearest Neighbor (KNN) vector search over virtual vec0 tables. Stores normalized 384-dimensional embeddings and executes SIMD-accelerated distance metrics.

Schema Definition

-- Master relational store
CREATE TABLE IF NOT EXISTS conversations (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id TEXT NOT NULL,
    speaker TEXT NOT NULL,
    content TEXT NOT NULL,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- Full-Text Search (FTS5) lexical index
CREATE VIRTUAL TABLE IF NOT EXISTS conversations_fts USING fts5(
    content,
    content='conversations',
    content_rowid='id'
);

-- sqlite-vec (vec0) virtual vector table
CREATE VIRTUAL TABLE IF NOT EXISTS conversations_vec USING vec0(
    embedding float[384]
);

Synchronized Trigger Operations

-- Automatic sync trigger for FTS5
CREATE TRIGGER IF NOT EXISTS conversations_ai AFTER INSERT ON conversations BEGIN
    INSERT INTO conversations_fts(rowid, content) VALUES (new.id, new.content);
END;

-- Automatic cleanup trigger for FTS5
CREATE TRIGGER IF NOT EXISTS conversations_ad AFTER DELETE ON conversations BEGIN
    INSERT INTO conversations_fts(conversations_fts, rowid, content) VALUES('delete', old.id, old.content);
END;

2. Dual-Path Retrieval via Reciprocal Rank Fusion (RRF)

Lexical search catches exact strings but misses synonyms; vector search captures conceptual meaning but struggles with exact symbol names. Combining both via Reciprocal Rank Fusion (RRF) produces a unified ranking without needing to normalize disparate raw score scales.

For any document d appearing in lexical results R_FTS or vector results R_vec, its RRF score is defined as:

S_{RRF} (d) = m \in FTS, vec \sum \frac{1}{k + r _{m} ( d )}

Where k = 60 is a standard smoothing constant, and r_m(d) is the 1-based ordinal rank of document d in search strategy m.

import sqlite3
import numpy as np

def hybrid_rrf_search(db_path: str, query_text: str, query_vector: list[float], top_k: int = 5) -> list[tuple[int, float]]:
    conn = sqlite3.connect(db_path)
    cur = conn.cursor()

    # Path 1: Lexical Search via FTS5 (BM25)
    cur.execute("""
        SELECT content_rowid FROM conversations_fts 
        WHERE conversations_fts MATCH ? 
        ORDER BY rank LIMIT 20
    """, (query_text,))
    fts_ranks = {row[0]: rank + 1 for rank, row in enumerate(cur.fetchall())}

    # Path 2: Semantic Search via sqlite-vec
    query_blob = np.array(query_vector, dtype=np.float32).tobytes()
    cur.execute("""
        SELECT rowid FROM conversations_vec 
        WHERE embedding MATCH ? 
        ORDER BY distance LIMIT 20
    """, (query_blob,))
    vec_ranks = {row[0]: rank + 1 for rank, row in enumerate(cur.fetchall())}

    # Step 3: Reciprocal Rank Fusion
    all_ids = set(fts_ranks.keys()).union(set(vec_ranks.keys()))
    rrf_scores = {}
    k = 60

    for doc_id in all_ids:
        score_fts = 1.0 / (k + fts_ranks[doc_id]) if doc_id in fts_ranks else 0.0
        score_vec = 1.0 / (k + vec_ranks[doc_id]) if doc_id in vec_ranks else 0.0
        rrf_scores[doc_id] = score_fts + score_vec

    sorted_results = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
    return sorted_results[:top_k]

3. Summarization Loops & Bounded State Management

To prevent multi-turn conversations from expanding into the mid-context dead zone, the backend executes an asynchronous Summarization Loop.

Active Window T_max Exceeded
  │
  ├─> [C_history (Turns 1..N-K)] ──> Summarizer ──> Compressed State S_t
  │
  └─> [C_recent (Turns N-K..N)] ─────────────────> Active Buffer

Conversation history is tracked in three state variables: active prompt context C_active, aging dialogue history C_history, and recent buffer C_recent.
When total dialogue tokens exceed a threshold T_max, C_history is sent to a background compression prompt that extracts key entities, open decisions, and active goals into a dense summary S_t.
S_t replaces C_history in working memory, maintaining an O(1) bounded context footprint while archiving raw dialogue turns to SQLite for long-term RRF retrieval.

4. Boundary-Weighted Prompt Assembly

Having retrieved relevant past memories via RRF and compressed active state via summarization loops, the final critical step is placement.

To bypass the mid-context dead zone, the prompt wrapper injects retrieved memories and summaries exclusively into the Primacy Zone and Recency Zone:

+-------------------------------------------------------------------------+
|                      BOUNDARY-WEIGHTED PROMPT SCHEMA                    |
+-------------------------------------------------------------------------+
| [PRIMACY ZONE]                                                          |
| - System Instructions & Persona Definition                              |
| - Top RRF Retrieved Memories (Rank 1..N)                                |
+-------------------------------------------------------------------------+
| [DEAD ZONE]                                                             |
| - [EXCLUDED / KEPT EMPTY TO PREVENT SOFTMAX DILUTION]                   |
+-------------------------------------------------------------------------+
| [RECENCY ZONE]                                                          |
| - Compressed Dialogue State Summary (S_t)                               |
| - Immediate User Query & Recent Chat Buffer                             |
+-------------------------------------------------------------------------+

By aligning critical context with the model's structural attention peaks, the prompt wrapper ensures maximum gradient influence during output generation.

Conclusion & Architecture Comparison

By replacing unbounded context expansion with an offline local memory pipeline, engineers achieve predictable recall, minimal latency, and strict data privacy.

Architectural Metric	Pure Context Expansion	Cloud Vector Database	Local SQLite Hybrid Engine
Compute Scaling	`O(N²)` quadratic KV overhead	`O(log M)` cloud search	`O(1)` FTS + SIMD KNN scan
Middle-Context Recall	Degrades factorially `O(1 / (H-1)!)`	Variable (depends on prompt layout)	High (boundary-weighted placement)
Deployment Footprint	Massive VRAM requirements	Heavy (SaaS keys, network daemons)	Zero-dependency (single .db file)
Exact Code/String Recall	Low (semantic attention)	Low (pure vectors miss exact symbols)	Exceptional (SQLite FTS5 BM25)
Data Privacy	Cloud inference dependent	Transmits data to external SaaS	100% Local / Offline

References & Further Reading

Liu, N. F., et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157–173.
Chowdhury, B. D. (2026). Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias. arXiv preprint arXiv:2603.10123.
Yi, Z., et al. (2024). Attention Basin: Why Contextual Position Matters in Large Language Models. In Proceedings of the Association for Computational Linguistics (ACL Anthology), 26086–26106.
Su, J., et al. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing, 568, 127063.
Garcia, A. (2024). sqlite-vec: A vector search SQLite extension that runs anywhere! GitHub Repository: asg017/sqlite-vec.
Aksan, C. (2026). Hybrid Search: Smart Search Architecture with FTS5 + Vector + RRF. Technical Report, dnomia-knowledge.
Zhou, Z., & Zhou, C. (2026). OScaR: Occam's Razor for Extreme KV Cache Quantization. Technical Review.

DEV Community: CV PRAJWAL GUPTHA