Knowledge and Memory Management: Directions 1-3 Finalization Record

We just closed the finalization record for Directions 1 through 3 in our knowledge and memory management subsystem. This covers the core pipeline: ingestion, storage, retrieval, and context integration. Here’s what that actually means for the architecture, why we made specific tradeoffs, and how to use it in your own stack.

The project has been iterating on how to decouple knowledge persistence from runtime memory while maintaining a unified query interface. Directions 1-3 form the foundation: a document store, a vector index, and a structured memory buffer that combines both. No more ad hoc caching or reinventing the retrieval loop. Everything lives behind a single KnowledgeGraph interface.

Direction 1: Raw Document Ingestion and Storage

We settled on a partitioned document store backed by a local SQLite database with a blob column for serialized content. Each document entry stores a UUID, source URI, raw text or bytes, a content hash, and a timestamp. The ingestion pipeline deduplicates by hash and runs through an optional extractor chain (e.g., PDF parser, markdown splitter, code chunker). The design decision is to separate storage from indexing entirely. The store is dumb—it only handles CRUD and metadata queries. This keeps the ingestion path simple and testable.

Direction 2: Vector Index with Filtered Search

Instead of building our own vector database, we wrapped existing infrastructure—Pinecone and a local FAISS fallback—behind an abstraction layer. The finalization record specifies a mandatory metadata filter set that must be packed into every upsert and query call. Each vector embedding carries a document UUID, chunk index, and a free-form tags map. This enables queries like “retrieve all chunks where module == 'networking' and version >= '2.0'” without scanning unrelated vectors.

The finalization also enforces a max-k retrieval of 50 with a similarity threshold of 0.65. Below that, the system returns an empty set rather than noisy garbage. We decided to precompute embeddings inside the ingestion pipeline and cache them to avoid recomputation during runtime. This makes the index stale until a sync command runs, which is acceptable for our latency requirements.

Direction 3: Structured Memory with Context Handles

This is the most important design direction. Memory is no longer a simple key-value store. Direction 3 introduces context handles—lightweight objects that tie a query to a specific knowledge window. Each handle holds a reference to the current conversation session, a list of retrieved document chunks, and a “cursor” that indicates what part of those chunks has already been consumed by the model.

The handle is serialized as JSON and stored inside the runtime memory buffer (an LRU cache in-process). When a request arrives, the system checks for an existing handle. If present, it resumes from the cursor instead of re-retrieving everything. If not, it creates a new handle from the query and the latest vector search results.

Here’s a short code example showing how a context handle is created and used:

from knowledge_memory import ContextHandle, KnowledgeGraph

graph = KnowledgeGraph(doc_store="sqlite:///docs.db", vector_index="faiss")
handle = ContextHandle(session="session-42")

# Retrieval only fetches chunks not yet consumed
results = graph.query("Deploy L4 load balancer", handle, max_chunks=5)

if results:
    # Each result knows its position in the handle's cursor
    for chunk in results:
        print(f"Processing chunk {handle.cursor}/{handle.total_chunks}")
        process_chunk(chunk.content, chunk.metadata)
        handle.advance()  # moves internal cursor

This avoided the classic bug where repeated queries in a conversation would return the same documents and cause the model to repeat itself. The cursor also enables progressive summarization: after every N chunks, the system can generate a partial summary and store it back into the handle.

Impact on Developer Experience

The new directions cut the configuration footprint by about half. Instead of wiring up separate database clients, embedding services, and memory stores, a developer instantiates a KnowledgeGraph and passes a context handle through the request flow. The handle carries all the state needed for multi-turn interactions.

One thing we still warn about: context handles are not thread-safe by default. The production wrapper uses a striped lock per session key. And the vector index is eventually consistent with the document store—if you delete a document, you must sync the index explicitly. The finalization record for Directions 1-3 doesn’t solve that yet; it’s on the roadmap for Direction 4.

For now, the foundation is solid. The ingestion pipeline is deterministic, the retrieval respects filters, and the memory model prevents information stalling. Experienced teams can adopt this pattern in a weekend.