Tobias Bjørkli - Pexels
Inside the technical aspects of VEKTOR’s improved agent memory
This article breaks down how our memory technology actually works in depth. If you look into some of the agent memory tools you’ll find a surprisingly thin layer: embed the text, store the vector, run a similarity search on recall. That’s not a criticism; it’s just where the industry grew from.
The interesting engineering happened after that baseline got built, in the layers nobody sees from the outside: what decides whether a new fact gets written at all, what happens to a memory nobody has touched in four months, and what stops the graph from slowly filling up with five slightly different phrasings of the same fact.
We went back into VEKTOR Slipstream’s architecture to walk through what’s really running under memory.store() and memory.recall(). This is a from-the-source technical breakdown of the six-layer pipeline, the reinforcement learning scorer sitting on top of it, the self-organizing link graph, the temporal reasoning engine, and the two-part security model that makes all of it defensible from a privacy standpoint.
The pipeline nobody talks about: what happens between store() and the disk?
Every memory system eventually has to answer the same question: given a new piece of information, what do you actually do with it. Some systems answer it in one step, embed it and append it. VEKTOR runs six distinct layers before a fact is considered settled, and each one exists because a specific failure mode showed up in practice.
All within a benchmarked 28 ms recall time, locally on a better-sqlite3 database in Node.js.
Layer 2, FadeMem differential decay. Every stored memory carries an importance score computed as a weighted blend: 0.4 times contextual relevance, 0.3 times frequency saturation, 0.3 times recency. That score feeds a decay function borrowed and adapted from a 2026 Alibaba and Peking University paper on differential memory fading (arXiv:2601.18642): strength decays as a stretched exponential, where lambda itself scales down as importance goes up, so genuinely important memories decay far slower than routine ones.
The beta exponent even differs by memory tier: 0.8 for long-term memories gives a sub-linear decay curve that's forgiving of gaps, while 1.2 for short-term memories gives a super-linear curve that drops off fast once something stops being relevant. There's also a causal protection term: a memory with important downstream consequences gets its decay dampened, on the logic that a fact several other facts depend on shouldn't fade just because nobody queried it directly in a while.
Layer 3, five-verdict conflict resolution. This is the layer that actually decides what happens to a new memory before it’s written, and it’s more nuanced than the classic add-update-delete-noop pattern most systems use.
VEKTOR’s conflict engine runs a cosine similarity check against recent memories, and only above a 0.72 threshold does it bother calling an LLM (batched up to ten pairs per call to stay within rate limits) to classify the relationship into one of five verdicts: COMPATIBLE, CONTRADICTORY, SUBSUMES, SUBSUMED, or NO_OP. Each verdict triggers a different write strategy.
COMPATIBLE memories both survive, but the existing one gets a small importance penalty proportional to how redundant it is. CONTRADICTORY triggers a trust-weighted suppression, and critically, a low-trust new memory cannot silently overrule a high-trust existing one; if the incoming fact’s trust score is under 80% of the existing memory’s, the system downgrades the verdict to COMPATIBLE instead of letting a shaky new input erase something solid.
SUBSUMES moves the more specific existing memory to cold storage rather than deleting it outright, and SUBSUMED does the reverse, reinforcing the broader existing memory and dropping the redundant new one.
Layer 4, fusion. Running as part of the idle-time REM cycle rather than inline with any user request, this layer clusters related memories from the past seven days using the same cosine similarity approach, and for any cluster of five or more, sends the full set to an LLM with instructions to produce one consolidated memory that preserves the temporal progression and unique facts across all of them.
The fused memory’s strength isn’t just an average, it’s the maximum strength across the cluster plus a small variance bonus, so a cluster of mostly-similar-but-one-different memories keeps more signal than pure averaging would destroy. The source memories move to cold storage rather than being deleted, connected to the new fused node through explicit fusion edges, so the provenance chain stays intact if you ever need to see what a summary was built from.
Layer 5, budgeted knapsack pruning. This is the fail-safe against graph bloat, and it runs as an actual knapsack optimization: memories get ranked by importance divided by the square root of their token count, not divided by raw token count, specifically so that dense, information-rich summaries aren’t penalized just for being longer. Each source type gets its own token and node budget, and anything that doesn’t fit gets moved to cold storage rather than hard-deleted, with pinned memories exempted entirely regardless of budget pressure.
Layer 6, additive reranking. At recall time, results aren’t ranked by similarity alone. The composite score is an explicit additive formula: 0.5 times similarity, plus 0.2 times strength, plus 0.15 times importance, plus 0.15 times a causal weight that gets boosted up to 1.5x for memories with important causal children. Additive, not multiplicative, on purpose, since a multiplicative formula lets any single low factor collapse the whole score, while additive keeps a memory in contention even if it’s weak on one dimension but strong on others.
6-layer memory path
What’s notable is that they exist as a coordinated pipeline, each one handling a specific failure mode the others don’t cover, rather than the more common pattern of bolting one clever technical tool onto a vector store and calling it memory.
Recall isn’t one search, it’s four running in parallel
The retrieval side runs what the codebase calls dual-channel recall, though by the current version it’s really four channels fused together with Reciprocal Rank Fusion.
Channel one is a standard semantic embedding search over stored content.
Channel two is BM25 full-text search via SQLite FTS5, catching exact terminology that semantic search sometimes paraphrases past.
Channel three, added to bridge a specific gap, embeds not just the stored content but an enriched version that includes the content’s “potential” context, closing the vocabulary gap between how something was written and how someone later asks about it.
Channel four is HyDE, Hypothetical Document Embeddings: before searching, the system asks a small fast model to write one hypothetical declarative sentence that would answer the query, then embeds that hypothetical answer and searches with it too.
That technique, adapted from Cloudflare’s HyperMem research, works because a hypothetical answer sits closer in embedding space to how facts are actually phrased than a question does, so it catches matches pure query embedding would miss.
All four channels get fused through RRF rather than a simple weighted average, and the fusion weighting itself is tunable per channel, not fixed.
On top of that fused score, layer 6’s additive reranking applies before anything reaches the caller. That’s five distinct scoring passes between a query going in and results coming out, which is a lot more machinery than “cosine similarity, top k” but is also the reason the system holds up on multi-hop and terminology-heavy queries where flat vector search alone tends to miss.
A learned layer sitting on top of the static one
Everything described so far uses fixed formulas, tuned constants baked into the architecture. Above that sits something genuinely adaptive: a reinforcement-learning memory scorer that logs which recalled memories actually got used in agent responses and trains a small logistic regression model on real usage patterns, gradually blending its learned importance signal into the static one.
The mechanism is simple by design. Every time memory is recalled, the system logs whether each result was actually used, alongside four features: the memory’s static importance, its recency (exponentially decayed based on time since last use), its usage frequency within a rolling 30-day window, and its confidence score.
Once at least ten usage samples exist, a logistic regression model trained via stochastic gradient descent starts producing a learned score, blended into the final ranking at a configurable ratio, 35% by default. The whole thing fails open by design: if the model isn’t trained yet, or the minimum sample threshold isn’t met, the system just falls back to the static formula with zero disruption.
This is a meaningfully different bet than most memory systems make. Instead of assuming the designers got the importance formula right on day one and leaving it fixed forever, the system is built to notice, over weeks of actual use, which memories the agent actually reaches for and nudge future ranking toward that observed pattern rather than the theoretical one.
The graph organizes itself while nobody’s watching
Separately from the six-layer write pipeline, there’s a background self-organization process modeled on the A-MEM and Zettelkasten note-linking pattern.
On every store call, an async agent generates keyword tags for the new memory, searches for semantically related existing memories, and asks an LLM to classify the relationship between each pair into one of five link types: SUPPORTS, EXTENDS, CONTRASTS, RELATED, or PREREQUISITE.
Those classified relationships get written as labeled edges into a dedicated link table, entirely separate from the causal and temporal edges in the main graph. For memories above an importance threshold, the system optionally goes a step further and synthesizes what the Zettelkasten tradition calls a permanent note, a short synthesis of what this memory means in the context of everything already linked to it.
Crucially, this runs fire-and-forget. The store call returns immediately; the linking and tagging happen asynchronously in the background, so a user or agent is never waiting on an LLM round trip just to save a fact. The system is explicitly designed to fail open on every error, meaning if the self-organization pass breaks for any reason, the underlying memory write already succeeded and nothing about core functionality degrades.
Confidence is a first-class, decaying signal
Separate again from importance and strength, every memory carries a confidence score from 0 to 1, starting at 1.0 on first write. It boosts by a small fixed amount when the same fact gets reinforced through a high-similarity write with no contradiction detected, and it decays, more aggressively, when a contradiction is detected against it.
A direct contradiction costs twice as much confidence as a softer supersession event. That score gets exposed directly in recall results, so a caller building on top of the memory layer can choose to filter or weight by confidence explicitly, treating a fact the system has contradicted once differently from one it’s reinforced five times.
Temporal reasoning that actually parses language, not just timestamps
A meaningful chunk of what makes long-context recall hard isn’t finding the right fact, it’s resolving what “two weeks ago” or “last Thanksgiving” actually means relative to when a conversation happened. VEKTOR’s temporal layer uses chrono-node, an NLP-based date parser, anchored to the session timestamp rather than the current moment, so relative expressions resolve correctly even when a memory is retrieved long after it was written.
The system pre-processes a handful of expressions chrono-node doesn’t natively catch (a fortnight, half a year, a couple of days) into forms it does handle, then falls back gracefully with no event date attached if the parser isn’t available at all, rather than failing the whole ingest. On the recall side, temporal queries run through explicit Julian-day SQL comparisons for anchoring, precedence, and interval-style questions, with an anti-join filter that excludes any node a later fact has explicitly superseded, so a query about “where do you live” doesn’t surface an address that’s already been corrected twice.
Two more layers most memory writeups never mention
Two smaller but distinctive pieces sit outside the core memory pipeline entirely, operating over code rather than conversation, which is worth mentioning because they show the same architecture being reused for a different problem.
One scans a project’s file structure on init and after significant changes, estimating token cost per file with different ratios for code, prose, and mixed formats, and writes the results as entity nodes into the same graph memory uses for everything else, giving an agent a persistent, queryable sense of a codebase’s shape without re-scanning it every session.
The other sits in front of every code write and checks it against known error patterns already recorded in the causal graph, using the same 0.72 similarity threshold as the core conflict engine, and if a new write matches a previously logged error signature, it warns rather than blocks.
That distinction matters: the system is built to never override the calling agent’s judgment, it only surfaces a pattern match and lets the agent decide, with an LRU cache bounding how often it re-checks near-identical content within a short window so it doesn’t slow down rapid multi-file edits.
Why none of this matters if the data isn’t yours
Six write-path layers, a learned reranking model, a self-organizing link graph, and a temporal parser are all pointless engineering if the underlying facts are sitting on infrastructure someone else controls.
This is where our Privacy Enhancing Technology approach isn’t a policy statement, it’s a structural constraint that shaped every layer above.
The graph lives in a SQLite file on the machine running it, by default, not in a hosted service. There’s no server-side copy of the memory graph for anyone to subpoena, misconfigure, or expose in a breach, because outside of the local file, it doesn’t exist.
That single fact is why most GDPR and CCPA obligations, data subject access requests, the right to erasure, processor agreements, mostly don’t apply in the first place: there’s no processor relationship to have, because nothing is being processed anywhere except the machine the user already controls. Deleting memory is a file operation performed directly by whoever holds it, not a support ticket routed through someone else’s backup rotation schedule.
Where memory does need to move, across a user’s own devices or within a team, encryption happens client-side before anything leaves the originating machine, so infrastructure in between never has an opportunity to see plaintext.
The other half of the security model: what happens when the agent acts
Memory security isn’t only about where facts sit at rest, it’s also about what’s allowed to write into that memory in the first place, and a memory-enabled agent is almost always also a tool-using one.
That’s the gap Faraday-Gate closes: a transparent proxy sitting between the agent and every tool server it talks to, doing three things before anything reaches memory or executes.
It hashes every tool schema on connection with SHA-256 and pins it, so if a tool server silently changes what a tool actually does between sessions, the hash mismatch is caught and the tool is blocked before the agent ever calls it, closing a specific supply chain window.
It tracks canary tokens and propagates taint through a call chain, so if something that shouldn’t be exposed shows up downstream, there’s a traceable path back to where it originated instead of a mystery.
And any action that crosses a risk threshold or deviates from the agent’s stated goal gets held in a queue rather than either firing automatically or being blocked outright, reviewed and approved or denied explicitly, so routine actions stay fast while genuinely risky ones wait for a human.
A compromised tool call is often exactly how bad data ends up written into a memory graph in the first place, so securing the store without securing what’s allowed to write to it only solves half the problem.
Where this is actually heading
The chart above sketches the shift in plain terms: retrieval accuracy across the field was already approaching a practical ceiling by 2025, most serious systems can find relevant text reasonably well now.
What’s still climbing steeply into 2026 and 2027 is everything downstream of retrieval: write-path curation, temporal reasoning, and autonomous consolidation, the parts that determine whether a memory graph stays trustworthy six months into continuous use rather than just on day one of a demo.
That’s consistent with what shows up across VEKTOR’s own architecture history: the six-layer pipeline, the RL scorer, and the self-organizing link graph were all added after the basic embed-and-search loop already worked, specifically because that basic loop degrades in ways that only show up over weeks of real usage, not in a single benchmark run.
Expect the next round of meaningful progress industry-wide to look less like better embeddings and more like better answers to a much less glamorous question: eight months from now, does this graph still make sense, or has it quietly filled up with contradictions nobody caught.
What to actually take from this if you’re working with agent memory
Decide the conflict resolution logic before picking an embedding model, since a fast vector index sitting on top of a write path that never deduplicates just gets you fast retrieval of an increasingly confused graph. Separate consolidation work from the request path entirely, so neither is compromising the other’s latency or thoroughness.
Treat confidence and importance as genuinely different signals rather than collapsing them into one score, since a fact can be important and simultaneously in doubt.
And treat where the data physically lives as a first-class architectural decision made at the start, not a deployment detail sorted out after the memory logic already exists, because retrofitting local-first onto a cloud-native design is a far harder rebuild than starting with the constraint already in place.
Decide who owns your physical memories and at what cost to get them in and out of the cloud.
VEKTOR Memory builds local-first, privacy-preserving persistent memory infrastructure for AI agents. Your data stays on your machine, by design, not by policy. Full technical documentation at vektormemory.com.
AI Agent
Vector Database
Generative Ai Tools
Cutting Edge Technology
Top comments (1)
I found the explanation of the six-layer memory pipeline in VEKTOR's architecture to be fascinating, particularly the discussion of the FadeMem differential decay and conflict resolution layers. The use of a weighted blend to compute importance scores and the adaptation of the differential memory fading paper from Alibaba and Peking University is a great example of how theoretical research can be applied to real-world problems. I'm curious to know more about how the causal protection term is implemented and how it affects the overall performance of the system. How do you handle cases where a memory has important downstream consequences, but its importance score is relatively low, and does this lead to any potential trade-offs in terms of memory retention versus recall accuracy?