Building a Scalable Agentic Memory Architecture from Scratch (No LangGraph, No AutoGen)

I've recently been working on building a custom Agentic AI framework completely from scratch—no LangGraph, no CrewAI, no AutoGen. The goal was to build a system with its own orchestrator, dynamic tool execution, and custom memory management.
Before diving in, we have to acknowledge the fundamental problem with AI engineering: LLM APIs are completely stateless. When you send a prompt to a model, it has zero memory of what you said 10 seconds ago. Every single time you call the API, you must resubmit the entire history of the conversation just so the agent has context. Designing how that context is packaged, compressed, and fed to the LLM is arguably the hardest part of building an autonomous agent.
In the early phases, the agent had a weak memory architecture. It relied entirely on a naive sliding window and aggressive output truncation to prevent context snowballing (where massive pip install logs or code outputs overflow the LLM's context window).
However, as testing progressed, I quickly ran into two massive bottlenecks:

1. Chat Amnesia: Because of the sliding window, the agent would literally "forget" critical architectural decisions made at the start of the chat once the conversation got long enough.
2. Project Pollution: When switching between different tasks or pulling context from older chats, the context would bleed over, confusing the agent about what project it was actually working on.

The Ultimate Motive: I needed a system that keeps the agent hyper-aware of the project context using the absolute minimum amount of tokens. Forcing the agent to re-read and re-analyze raw logs over and over again burns expensive API tokens and wastes time. I needed a highly token-efficient memory structure.
To overcome this, I designed and built a Tiered Memory Architecture. But before diving into the memory, it's important to understand the physical hierarchy of how data is stored.

The Hierarchy: Workspaces, Projects, and Chats

To keep everything organized and prevent context-bleeding, I structured the physical environment into a strict hierarchy:

1. Root Workspace Folder: The master directory that holds all physical project files and the central SQLite database.
2. Projects (Workspaces): Individual sandboxed projects (like the Tic-Tac-Toe game or a React app). Each project has a unique workspace_id.
3. Chats (Threads): Inside every project, there are multiple isolated chat threads (e.g., one chat for building the backend, a completely separate chat for debugging the UI). This strict structural hierarchy is what makes the Tiered Memory Architecture possible. Here is how the memory works under the hood.

The Solution: A Tiered Memory Architecture (G(P) and S(P)) and an Active Sliding Window

To fix amnesia and maximize token efficiency, I split the agent's memory into distinct isolated summary layers. The heavy-reasoning agent (like GPT-4o) never just gets a raw list of messages. Instead, it receives a perfectly balanced, low-token context payload made of three core components:

1. Global Memory — G(P) (Static Summary)

G(P) represents the immutable Project Blueprint. When a workspace is initialized, G(P) is defined. It tells the agent the fundamental truth of the environment (e.g., "This is a Connect Four game built in Python using Tkinter"). Its Role: It acts as the static anchor. No matter how deep into a debugging rabbit hole the agent goes, G(P) ensures it never forgets the overarching goal of the project.

2. Session Memory — S(P) (Dynamic Summary)

S(P) represents the dynamic, Thread-Specific Context. Every individual chat thread has its own S(P). It is a highly compressed, operational summary of exactly what has happened in this specific conversation (e.g., "We just built the UI grid, but we are currently debugging an absolute import error in basic_gui.py"). Its Role: It solves Chat Amnesia and Token Burning. By constantly updating this summary, the agent remembers the journey of the chat without needing thousands of tokens of raw message history.

3. The Active Sliding Window (Immediate Context)

While G(P) and S(P) provide the high-level summaries, the heavy-reasoning LLM still needs to know exactly what just happened 5 seconds ago to write code effectively. This is where the Active Sliding Window comes in. It contains only the last n raw messages (usually just the immediate prompts and recent tool outputs). Its Role: It gives the Heavy Agent the immediate, line-by-line syntax and terminal errors it needs to solve the current step, without letting those massive logs pollute the permanent memory.

The Memory Lifecycle: Folding and Restoring Context

Updating S(P) (Background Folding): S(P) is not static; it evolves. When a chat thread hits a specific threshold (e.g., n prompts in the sliding window), an async background task triggers. It slices off the oldest raw messages, sends them to a smaller, cheaper SLM (like ministral-3b), and generates an updated S(P) summary. The raw messages are then archived to ChromaDB, keeping the active sliding window incredibly small and token-efficient.
Restoring Context: If a user exits a chat, the raw sliding window for that session is essentially dropped from active processing. But because S(P) was continuously updated and saved in SQLite, the memory isn't lost. When the user opens the old chat again, the agent instantly regains full context by looking at the combination of the static G(P) and the dynamic S(P) together.

Handling Cross-Project Isolation

One of the hardest challenges was allowing the agent to reference older chats without triggering Project Pollution.
To solve this, I implemented a strict isolation protocol. S(P) is strictly bound to a specific chat_id inside SQLite. If I open a cross-project chat to borrow code, the framework applies a Read-Only Firewall. The agent can use semantic search to read the old chat's logic, but the old chat's memory state is completely sandboxed from the active workspace's G(P).

Implementation Details: SQLite, ChromaDB, Async Locks

The actual flow of the memory system is highly asynchronous to keep the agent fast. Here is the technical stack:

1. The Context Assembly

When the user sends a prompt, the Memory Manager queries ChromaDB (using Nomic embeddings) to fetch the relevant semantic chunks from past conversations. It then stitches together G(P) + S(P) + RAG chunks + the recent raw messages into a single dense payload for the Orchestrator.

2. Solving Race Conditions with Symmetrical Locks

Because the Heavy Agent is constantly writing new tool outputs to the SQLite database while the Background Folder is simultaneously trying to read, summarize, and truncate that exact same database, I immediately ran into severe race conditions causing data corruption.
To fix this, I implemented a Symmetric Memory Lock (an asyncio.Lock mapped per chat_id). Before the background folder truncates the database, it acquires the lock. It ensures an Atomic Truncation—the SQLite sliding window is only truncated if both the SLM summarization and the ChromaDB embedding succeed. If either fails, the lock is released and the sliding window is left intact to prevent data loss.

A Humble Reflection on Industry Standards

When I set out to build this memory hierarchy, my only goal was to solve the severe context limitations and token-burning I was experiencing locally. I didn't use any external frameworks as reference.
However, after finishing the architecture, I was incredibly surprised (and validated!) to learn that this exact setup closely mirrors the gold standard used by major AI researchers today. The concept of Tiered Memory (static core memory vs. dynamic recall memory) is the foundation of the famous MemGPT paper from UC Berkeley. Furthermore, combining a static project blueprint with thread-isolated summaries is precisely how Anthropic architected Claude Projects.
Realizing that I had independently converged on the same architectural conclusions as top-tier AI labs while building my own framework from scratch was a deeply rewarding moment as an engineer. It proves that when you focus deeply on solving a core engineering bottleneck (like token efficiency), the right architecture naturally reveals itself.