I collected a week of design notes on agent memory, built a small store in .NET to test three of them, and found two leaks I would have shipped
This is the fourth piece in a row I’ve written about agent memory. In “Your Agent’s Memory Is Probably Wrong” I argued that most memory bugs are lifecycle bugs. In “Agent Memory After the Hype” I walked back the grep-beats-graph framing and kept supersession and TTL. In “Five Memory Types, One Store” I merged and split typed stores and measured what broke.
All three of those pieces were about one question: does the agent get the right fact back? This week the reading kept pushing on two questions I had mostly skipped. Who is allowed to get the fact back? And when someone asks for a fact to be gone, how do you prove it is gone?
So this piece has three verbs in the title. Learn, because two of the sources this week are about agents that improve from their own runs. Forget, because one of them is about the right to be forgotten and another is about facts that silently go stale. Permissioned, because the most concrete design document I read was a vendor saying, more or less, that permissions belong in the data layer and not in the prompt.
I’ll go through what I read first, with every number labelled by who produced it. Then I’ll show the store I built, the two scenarios I ran against it, and what failed. The failures are the useful part.
How to read the numbers in this piece
Nearly every figure below comes from a vendor, a newsletter, or an author describing their own project. I label each one. None of them were independently reproduced by me. The only numbers I can stand behind are the ones from my own small test at the end, and those come with their own caveats, which I’ll spell out when we get there.
What I read this week
1. Atlassian’s Teamwork Graph: four design rules and one vendor benchmark
I first saw this in The Code newsletter on 15 October, which summarised Atlassian’s Teamwork Graph as following four design rules:
- Link tickets, docs and decisions to code, so the context an agent pulls is connected rather than a pile of search hits.
- Expose the graph through MCP or a CLI, so any agent can query it.
- Enforce permissions at the data layer, not in the agent.
- Write every session back, so the graph learns from what agents did.
I went to Atlassian’s own posts to see how much of that holds up in their words. The Team ’26 announcement describes the graph as a map linking people, goals, code and content across Atlassian and connected apps, and says that data coming in through custom connectors keeps its access controls, so AI respects who can see what. The CLI general availability post on the Atlassian community is more concrete: separate read, write and delete scopes (delete is its own scope, so an agent can write without being able to delete), OAuth 2.1 with short-lived credentials, admin policies for scopes, and every CLI call landing in the audit log.
The fourth rule is the one I couldn’t find stated that way. The CLI is clearly not read-only: you can create relationships, update work items and push context back into the graph. But “every session is written back” is the newsletter’s framing, not something I found in the two Atlassian posts I read. I’m treating it as a reasonable reading, not as a documented guarantee.
Then the number. Atlassian says that grounding agent responses in Teamwork Graph data produced 44% more accurate answers while using 48% fewer tokens. Both posts attribute this to their own benchmarks; the community post calls it an internal study. Neither post describes the tasks, the models, the baseline, the sample size or how accuracy was scored. The only comparison mentioned is against agents that query each product separately. So: vendor number, internal benchmark, no published method. I find it plausible in direction (pre-joined context should beat an agent stitching five API calls together) and I have no way to judge the size.
The rule I actually took away is the third one. If permissions live in the data layer, then a memory store is part of the data layer, and my memory stores have never had a permission model at all. That became the second scenario below.
2. Jaroslaw Wasowski: the right to be forgotten in agent memory
Wasowski’s piece in Level Up Coding starts from a simple observation that made me uncomfortable: closing an erasure request by deleting the conversation log does not delete what the agent extracted from that conversation. He points to OpenAI’s 2024 statement that deleting a ChatGPT chat doesn’t delete the memories saved from it. A fact pulled from one conversation can sit in the memory store, in an embedding, in a summary and in a backup, and a real erasure has to reach all of them.
The article goes on to break the memory lifecycle into five write-time decisions and proposes an erasure test you can run against your own system. I only got through the opening section, so I won’t pretend to summarise his five decisions. The framing alone was enough to change my plan: I wanted a delete path that produces evidence, not a delete path that returns a row count.
3. Miodrag Cekikj: a course compiler with a gate that refuses unsupported claims
Cekikj built a system that compiles adaptive course material from sources, and the part I care about is the gate. Generated modules go through a submit_module step, and a module that contains claims not supported by the source material is rejected rather than stored. That is a write-time check: bad content never becomes durable content.
He reports that after adding the gate, 594 of 645 claims were judged supported, which is 92%, against 77% in an earlier version. He is careful about it himself: the comparison is indicative, not a controlled experiment, and the judge model comes from the same vendor as the model doing the generating. I’d add the obvious concern that follows from that: a judge from the same family may share blind spots with the generator, so 92% is “the judge agreed with itself more often”, which is still useful but is not ground truth.
Why it belongs in a memory piece: a memory store is also a compiler. Every write is a claim. Most memory systems I’ve seen, including mine, accept any write the agent makes. A gate like submit_module is the cheapest place to stop a wrong fact, because once it is written it gets retrieved, consolidated and repeated.
4. Reef: an agent that learns from its own runs, with a receipt
Kristopher Dunham wrote about Reef, an open source project (github.com/Human-Agent-Society/reef) built around a loop of serve, observe, grow and commit. The inference server serves the agent, observes successes and failures, grows an improvement from them (new weights if you have GPUs, better rules and skills if you don’t), and commits the change. Each commit carries a receipt header, so you can trace a behaviour change back to the evidence that produced it.
I like the receipt idea a lot. It is the learning-side equivalent of the provenance I keep asking for in memory: if the agent changed, I want to know why.
Two caveats, and they matter. First, the default evaluation in the loop is only an echo health check. Out of the box, “the grown version passed evaluation” means “it still responds”, not “it got better”. Second, the SAO numbers in the write-up are reported by the project itself. With a default eval that weak, I wouldn’t put any weight on improvement numbers until someone plugs in a real task-level evaluation and reports against that.
5. AutoResearch at production scale (arXiv 2609.30541)
Gowtham Boyina’s write-up is titled around Karpathy’s 630-line AutoResearch script running at “Amazon scale”. The paper it covers is “AutoResearch at Production Scale: Failure Modes and a Multi-Agent Framework” by Aparajith Chandran, Juwon Kim, Saurav Jha, Pablo Castells and Florian Hottier. I read the abstract, and it is worth being precise about what it says: twelve weeks running an autonomous research loop on two independently developed representation-learning systems for a book recommendation pipeline, more than 220 experiments, and five failure modes.
The abstract says book recommendation. It does not say Amazon. The Medium headline does. I’d treat the Amazon framing as the write-up’s, not the paper’s.
The five failure modes, as the paper names them:
- infrastructure fragility
- agent memory decay
- search-direction stagnation
- iteration-cost asymmetry
- metric fixation
The paper proposes a three-part scaffolding (prevent, persist, redirect) and reports a 1.82x Recall@6 lift over hand-tuned baselines. Boyina’s article gives the underlying figures as 3.79% to 6.90%, which is consistent with 1.82x. These are the authors’ reported results on their own pipeline.
The failure mode that matters here is the second one. A research agent running for twelve weeks forgets what it already tried. That is the same bug as an agent recommending an old pricing plan: memory that doesn’t track what replaced what. The “persist” part of their scaffolding is, in effect, a memory store with structure, and it shows up in a paper about research automation, not a paper about memory.
6. The Codex memory config hack, checked against the source
The Code newsletter also passed along a Codex tip: you can pin which models the memory pipeline uses, through extract_model and consolidation_model. My brief to myself was to check the names against OpenAI's config reference before repeating them. The docs page redirected somewhere my tools couldn't reach, so I checked the source instead. In the openai/codex repository, as of the commit I pulled on 18 October 2026, both keys exist under a [memories] table, documented in codex-rs/config/src/types.rs as "Model used for thread summarisation" and "Model used for memory consolidation". The feature itself sits behind a memories flag in [features], which is marked stable and off by default.
Two details from the source were more interesting than the hack itself. When extract_model is unset, Codex falls back to whatever model the provider marks as its preferred extraction model, which is exactly why people noticed memory calls going to models they hadn't chosen. And there is a key called disable_on_external_context: when true, a thread that pulled in external context (it also accepts the alias no_memories_if_mcp_or_web_search) is marked as polluted, which as I read the source keeps it out of memory generation. That is a permission decision hiding in a config file. If your agent reads a Jira ticket through MCP, should a summary of that ticket end up in a memory that outlives the session? Codex gives you a switch for it. Most homegrown stores, mine included, don't.
Here is a working config that keeps the whole memory pipeline on a local model through Codex’s built-in Ollama provider (ollama pull gpt-oss:20b first):
# ~/.codex/config.toml
model_provider = "ollama"
model = "gpt-oss:20b"
[features]
memories = true
[memories]
extract_model = "gpt-oss:20b" # per-thread extraction
consolidation_model = "gpt-oss:20b" # global consolidation
disable_on_external_context = true # threads that used MCP or web search do not become memories
max_unused_days = 30 # the source default; as I read it, memories unused this long age out
I haven’t measured extraction quality with a 20B local model against a hosted one. I’d expect it to be worse. What you get in exchange is that the consolidation step, the step that merges facts across threads, runs on your machine.
7. Sakshi: stale facts need edges, not distances
Sakshi’s piece on choosing between graph memory and vector memory opens with a story I recognised immediately: an agent kept recommending a user’s old pricing plan after the user had switched. Retrieval tests passed. Embeddings were fresh. The problem was that vector memory has no concept of “this fact replaced that fact”. Her answer, in the graph camp, is to put time on the relationship: an edge carries valid_from and valid_until, and "current" means the edge whose valid_until is empty.
I agree with the framing and I tested it below, in a flattened form (two columns on a table instead of edges in a graph). Her piece also contains benchmark numbers comparing approaches. I’m not reusing them: they are secondhand as far as I can tell, and I couldn’t trace them to a run I could inspect. The same goes for her “5 .md files every agent needs” piece: the idea of a MEMORY.md the agent reads at startup is fine, but the claims about it are unmeasured, and a markdown file has exactly the same stale-fact problem unless something owns supersession.
The design rules I took into the build
Pulling those together, this is what I wanted the store to do:
- Typed entries. Every memory has a kind (Fact, Preference, Decision, Episode, Summary), a subject and a key, so “replace” has something to match on.
- Supersession, not deletion. A new value closes the old one with ValidUntil and points at its replacement. History stays, "current" is a query.
- TTL. Episodes like “deploy freeze until Friday” expire on their own.
- Permission filter before retrieval. The visibility rule runs in SQL, before a single vector is loaded or scored. Nothing the user can’t see is ever ranked.
- Delete by subject, with proof. After deletion, show that the rows are gone, the vectors are gone, nothing orphaned remains, and the text is not sitting in the database file.
The build: .NET, EF Core, SQLite, local Ollama embeddings
Setup
# local embedding model
ollama pull nomic-embed-text
# project
dotnet new console -n MemoryLab -f net10.0
cd MemoryLab
dotnet add package Microsoft.EntityFrameworkCore.Sqlite --version "10.0.*"
If you’d rather keep Ollama in a container:
docker run -d --name ollama -p 11434:11434 -v ollama:/root/.ollama ollama/ollama
docker exec ollama ollama pull nomic-embed-text
Then replace the generated files with the four below and run dotnet run. OLLAMA_URL and EMBED_MODEL environment variables override the defaults (http://localhost:11434 and nomic-embed-text).
The project file
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="10.0.*" />
</ItemGroup>
</Project>
Model.cs: typed entries, a separate embeddings table, team membership
The embeddings live in their own table on purpose. It makes “prove the vectors are gone” a query against a specific table instead of a hope.
using Microsoft.EntityFrameworkCore;
namespace MemoryLab;
public enum MemoryKind { Fact, Preference, Decision, Episode, Summary }
public enum MemoryScope { Private, Team }
public class MemoryEntry
{
public long Id { get; set; }
public required string OwnerUserId { get; set; }
public MemoryScope Scope { get; set; }
public string? TeamId { get; set; }
public required string Subject { get; set; } // "user:alice", "service:staging"
public MemoryKind Kind { get; set; }
public required string Key { get; set; } // "db.engine"
public required string Value { get; set; }
public DateTime ValidFrom { get; set; }
public DateTime? ValidUntil { get; set; } // null = current
public long? SupersededById { get; set; }
public DateTime? ExpiresAt { get; set; } // TTL, null = no expiry
public DateTime CreatedAt { get; set; }
public MemoryEmbedding? Embedding { get; set; }
}
public class MemoryEmbedding
{
public long EntryId { get; set; }
public required string Model { get; set; }
public int Dim { get; set; }
public required byte[] Vector { get; set; }
public MemoryEntry Entry { get; set; } = null!;
}
public class TeamMember
{
public required string UserId { get; set; }
public required string TeamId { get; set; }
}
public class ErasureRecord
{
public long Id { get; set; }
public required string Subject { get; set; }
public int Entries { get; set; }
public int Vectors { get; set; }
public DateTime At { get; set; }
}
public class MemoryDb(DbContextOptions<MemoryDb> options) : DbContext(options)
{
public DbSet<MemoryEntry> Entries => Set<MemoryEntry>();
public DbSet<MemoryEmbedding> Embeddings => Set<MemoryEmbedding>();
public DbSet<TeamMember> TeamMembers => Set<TeamMember>();
public DbSet<ErasureRecord> Erasures => Set<ErasureRecord>();
protected override void OnModelCreating(ModelBuilder b)
{
b.Entity<MemoryEntry>(e =>
{
e.Property(x => x.Scope).HasConversion<string>();
e.Property(x => x.Kind).HasConversion<string>();
e.HasIndex(x => new { x.Subject, x.Kind, x.Key, x.ValidUntil });
e.HasIndex(x => x.OwnerUserId);
});
b.Entity<MemoryEmbedding>(e =>
{
e.HasKey(x => x.EntryId);
e.HasOne(x => x.Entry).WithOne(x => x.Embedding)
.HasForeignKey<MemoryEmbedding>(x => x.EntryId)
.OnDelete(DeleteBehavior.Cascade);
});
b.Entity<TeamMember>().HasKey(x => new { x.UserId, x.TeamId });
}
}
Embedder.cs: local Ollama
nomic-embed-text expects search_query: and search_document: prefixes, so the embedder adds them. Note the in-process cache. I'll come back to it, because it turned out to be an erasure problem of its own.
using System.Net.Http.Json;
using System.Runtime.InteropServices;
namespace MemoryLab;
public interface IEmbedder
{
string Model { get; }
Task<float[]> EmbedAsync(string text, bool isQuery, CancellationToken ct = default);
}
// Local Ollama: `ollama pull nomic-embed-text`, server on http://localhost:11434
public sealed class OllamaEmbedder(HttpClient http, string model = "nomic-embed-text") : IEmbedder
{
public string Model => model;
private readonly Dictionary<string, float[]> _cache = new();
public async Task<float[]> EmbedAsync(string text, bool isQuery, CancellationToken ct = default)
{
// nomic-embed-text expects task prefixes
var input = (isQuery ? "search_query: " : "search_document: ") + text;
if (_cache.TryGetValue(input, out var hit)) return hit;
var res = await http.PostAsJsonAsync("/api/embed", new { model, input }, ct);
res.EnsureSuccessStatusCode();
var body = await res.Content.ReadFromJsonAsync<EmbedResponse>(cancellationToken: ct)
?? throw new InvalidOperationException("empty embed response");
var v = Normalize(body.Embeddings[0]);
_cache[input] = v;
return v;
}
private static float[] Normalize(float[] v)
{
var n = MathF.Sqrt(v.Sum(x => x * x));
return n == 0 ? v : v.Select(x => x / n).ToArray();
}
private sealed record EmbedResponse(float[][] Embeddings);
}
public static class Vec
{
public static byte[] Pack(float[] v) => MemoryMarshal.AsBytes(v.AsSpan()).ToArray();
public static float[] Unpack(byte[] b) => MemoryMarshal.Cast<byte, float>(b).ToArray();
public static float Cosine(float[] a, float[] b)
{
float d = 0, na = 0, nb = 0;
for (var i = 0; i < a.Length; i++) { d += a[i] * b[i]; na += a[i] * a[i]; nb += b[i] * b[i]; }
return na == 0 || nb == 0 ? 0 : d / (MathF.Sqrt(na) * MathF.Sqrt(nb));
}
}
MemoryStore.cs: supersession, TTL, the permission filter, consolidation and erasure
The StoreOptions record carries the switches I used to reproduce my first version (v1) and the fixed one (v2), so both are visible in one file.
using System.Data.Common;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Diagnostics;
namespace MemoryLab;
public record StoreOptions(
bool ExactSupersession = true,
bool ScopeAwareSupersession = true, // my v1 had this false
float? SemanticSupersedeAt = null, // I do not recommend turning this on, see the article
bool PreFilter = true, // v1: rank everything, then filter
bool ConsolidatePerScope = true, // v1: one summary from every scope
float MinScore = 0.0f);
public record Hit(long Id, string Value, float Score);
public record ErasureProof(
int RowsLeft, int OrphanVectors, List<long> LiveMentions,
List<float> NearestCosine, Dictionary<string, List<string>> BytesOnDisk);
// secure_delete is a per-connection setting, so it has to be applied every time EF opens one
public sealed class SqlitePragmas(bool secureDelete) : DbConnectionInterceptor
{
public override async Task ConnectionOpenedAsync(DbConnection c, ConnectionEndEventData d, CancellationToken ct = default)
{
await using var cmd = c.CreateCommand();
cmd.CommandText = $"PRAGMA foreign_keys=ON; PRAGMA secure_delete={(secureDelete ? "ON" : "OFF")};";
await cmd.ExecuteNonQueryAsync(ct);
}
public override void ConnectionOpened(DbConnection c, ConnectionEndEventData d)
{
using var cmd = c.CreateCommand();
cmd.CommandText = $"PRAGMA foreign_keys=ON; PRAGMA secure_delete={(secureDelete ? "ON" : "OFF")};";
cmd.ExecuteNonQuery();
}
}
public sealed class MemoryStore(MemoryDb db, IEmbedder embedder, TimeProvider clock, StoreOptions opt)
{
private DateTime Now => clock.GetUtcNow().UtcDateTime;
// ---------------------------------------------------------------- visibility
// The one rule every read goes through. Applied in SQL, before any vector is loaded.
public IQueryable<MemoryEntry> Visible(string userId) =>
db.Entries.Where(e =>
e.OwnerUserId == userId ||
(e.Scope == MemoryScope.Team &&
db.TeamMembers.Any(m => m.UserId == userId && m.TeamId == e.TeamId)));
// ---------------------------------------------------------------- write
public async Task<(long Id, List<long> Superseded)> WriteAsync(
string owner, MemoryScope scope, string subject, MemoryKind kind, string key, string value,
string? team = null, TimeSpan? ttl = null, CancellationToken ct = default)
{
var now = Now;
var vec = await embedder.EmbedAsync(value, isQuery: false, ct);
var entry = new MemoryEntry
{
OwnerUserId = owner, Scope = scope, TeamId = team, Subject = subject, Kind = kind,
Key = key, Value = value, ValidFrom = now, CreatedAt = now,
ExpiresAt = ttl is null ? null : now + ttl.Value,
Embedding = new MemoryEmbedding { Model = embedder.Model, Dim = vec.Length, Vector = Vec.Pack(vec) }
};
db.Entries.Add(entry);
await db.SaveChangesAsync(ct);
var olds = new List<MemoryEntry>();
if (opt.ExactSupersession)
{
var q = db.Entries.Where(e => e.Subject == subject && e.Kind == kind && e.Key == key
&& e.ValidUntil == null && e.Id != entry.Id);
if (opt.ScopeAwareSupersession)
// a private note may only replace the writer's own private note;
// a team fact may only replace a team fact of the same team
q = q.Where(e => e.Scope == scope && e.TeamId == team &&
(e.Scope == MemoryScope.Team || e.OwnerUserId == owner));
olds.AddRange(await q.ToListAsync(ct));
}
if (opt.SemanticSupersedeAt is float threshold)
{
var near = await db.Entries
.Where(e => e.Subject == subject && e.Kind == kind && e.ValidUntil == null
&& e.Id != entry.Id && e.Scope == scope && e.TeamId == team)
.Include(e => e.Embedding)
.ToListAsync(ct);
olds.AddRange(near.Where(e => !olds.Contains(e) &&
Vec.Cosine(vec, Vec.Unpack(e.Embedding!.Vector)) >= threshold));
}
foreach (var o in olds) { o.ValidUntil = now; o.SupersededById = entry.Id; }
await db.SaveChangesAsync(ct);
return (entry.Id, olds.Select(o => o.Id).ToList());
}
// ---------------------------------------------------------------- read
public async Task<List<Hit>> SearchAsync(string userId, string query, int k = 3, CancellationToken ct = default)
{
var now = Now;
var qv = await embedder.EmbedAsync(query, isQuery: true, ct);
IQueryable<MemoryEntry> pool = opt.PreFilter ? Visible(userId) : db.Entries;
var rows = await pool
.Where(e => e.ValidUntil == null && (e.ExpiresAt == null || e.ExpiresAt > now))
.Select(e => new { e.Id, e.Value, e.Embedding!.Vector })
.ToListAsync(ct);
var top = rows.Select(r => new Hit(r.Id, r.Value, Vec.Cosine(qv, Vec.Unpack(r.Vector))))
.Where(h => h.Score >= opt.MinScore)
.OrderByDescending(h => h.Score).Take(k).ToList();
if (opt.PreFilter) return top;
// v1 behaviour: filter after ranking. Nothing leaks here, but the user gets fewer than k results.
var ids = top.Select(t => t.Id).ToList();
var allowed = await Visible(userId).Where(e => ids.Contains(e.Id)).Select(e => e.Id).ToListAsync(ct);
return top.Where(t => allowed.Contains(t.Id)).ToList();
}
// Follow superseded_by to the live version. Each hop is checked; an invisible hop stops the walk.
public async Task<string?> CurrentVersionAsync(string userId, long entryId, CancellationToken ct = default)
{
var seen = new HashSet<long>();
long? id = entryId;
while (id is long cur && seen.Add(cur))
{
var row = await Visible(userId).Where(e => e.Id == cur)
.Select(e => new { e.Value, e.SupersededById }).FirstOrDefaultAsync(ct);
if (row is null) return null;
if (row.SupersededById is null) return row.Value;
id = row.SupersededById;
}
return null;
}
// ---------------------------------------------------------------- consolidation
public async Task<long?> ConsolidateAsync(string subject, string team, CancellationToken ct = default)
{
var now = Now;
var q = db.Entries.Where(e => e.Subject == subject && e.ValidUntil == null && e.Kind != MemoryKind.Summary
&& (e.ExpiresAt == null || e.ExpiresAt > now));
if (opt.ConsolidatePerScope)
q = q.Where(e => e.Scope == MemoryScope.Team && e.TeamId == team); // private notes never feed a team summary
var values = await q.OrderBy(e => e.Id).Select(e => e.Value).ToListAsync(ct);
if (values.Count == 0) return null;
var (id, _) = await WriteAsync("system", MemoryScope.Team, subject, MemoryKind.Summary, "summary",
string.Join(" | ", values), team, ct: ct);
return id;
}
// ---------------------------------------------------------------- erasure
public async Task<(int Entries, int Vectors)> DeleteSubjectAsync(string subject, CancellationToken ct = default)
{
await using var tx = await db.Database.BeginTransactionAsync(ct);
var ids = await db.Entries.Where(e => e.Subject == subject).Select(e => e.Id).ToListAsync(ct);
// Do not rely on the FK cascade alone: delete vectors explicitly, then rows.
var vectors = await db.Embeddings.Where(v => ids.Contains(v.EntryId)).ExecuteDeleteAsync(ct);
var entries = await db.Entries.Where(e => ids.Contains(e.Id)).ExecuteDeleteAsync(ct);
db.Erasures.Add(new ErasureRecord { Subject = subject, Entries = entries, Vectors = vectors, At = Now });
await db.SaveChangesAsync(ct);
await tx.CommitAsync(ct);
// Rewrite the file so freed pages are gone, then fold the WAL back in.
// VACUUM in WAL mode lands in the -wal file; without the checkpoint the old pages stay in the main file.
await db.Database.ExecuteSqlRawAsync("VACUUM;", ct);
await db.Database.ExecuteSqlRawAsync("PRAGMA wal_checkpoint(TRUNCATE);", ct);
return (entries, vectors);
}
public async Task<ErasureProof> ProveGoneAsync(string subject, string[] needles, string dbPath, CancellationToken ct = default)
{
var rowsLeft = await db.Entries.CountAsync(e => e.Subject == subject, ct);
var orphans = await db.Embeddings.CountAsync(v => !db.Entries.Any(e => e.Id == v.EntryId), ct);
var mentions = new List<long>();
foreach (var n in needles)
mentions.AddRange(await db.Entries.Where(e => EF.Functions.Like(e.Value, $"%{n}%"))
.Select(e => e.Id).ToListAsync(ct));
var vecs = await db.Embeddings.Select(v => v.Vector).ToListAsync(ct);
var nearest = new List<float>();
foreach (var n in needles)
{
var q = await embedder.EmbedAsync(n, isQuery: false, ct);
nearest.Add(vecs.Count == 0 ? 0 : vecs.Max(b => Vec.Cosine(q, Vec.Unpack(b))));
}
var bytes = new Dictionary<string, List<string>>();
foreach (var suffix in new[] { "", "-wal", "-journal" })
{
var path = dbPath + suffix;
if (!File.Exists(path)) continue;
byte[] raw;
using (var fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (var ms = new MemoryStream()) { await fs.CopyToAsync(ms, ct); raw = ms.ToArray(); }
bytes[suffix == "" ? "db" : suffix] = needles
.Where(n => raw.AsSpan().IndexOf(System.Text.Encoding.UTF8.GetBytes(n)) >= 0).ToList();
}
return new ErasureProof(rowsLeft, orphans, mentions.Distinct().ToList(), nearest, bytes);
}
}
Program.cs: the two scenarios
using Microsoft.EntityFrameworkCore;
using MemoryLab;
using static MemoryLab.MemoryKind;
using static MemoryLab.MemoryScope;
var http = new HttpClient { BaseAddress = new Uri(Environment.GetEnvironmentVariable("OLLAMA_URL") ?? "http://localhost:11434") };
var embedder = new OllamaEmbedder(http, Environment.GetEnvironmentVariable("EMBED_MODEL") ?? "nomic-embed-text");
Directory.CreateDirectory("runs");
// ---------------------------------------------------------------- stale-fact scenario
foreach (var (label, opt) in new[]
{
("raw", new StoreOptions(ExactSupersession: false)),
("key", new StoreOptions()),
("key+sem0.80", new StoreOptions(SemanticSupersedeAt: 0.80f)),
})
{
var (store, clock, _) = await Fresh($"stale_{label}", opt);
var oldIds = new List<long>(); var newIds = new List<long>(); var nbIds = new List<long>();
foreach (var u in Data.Updates) { oldIds.Add((await store.WriteAsync("alice", Team, u.Subject, u.Kind, u.Key, u.Old, "payments")).Id); clock.Advance(TimeSpan.FromHours(1)); }
foreach (var n in Data.Neighbours) { nbIds.Add((await store.WriteAsync("alice", Team, n.Subject, n.Kind, n.Key, n.Value, "payments")).Id); clock.Advance(TimeSpan.FromHours(1)); }
clock.Advance(TimeSpan.FromDays(14));
var wronglySuperseded = new List<long>();
for (var i = 0; i < Data.Updates.Length; i++)
{
var u = Data.Updates[i];
var (id, olds) = await store.WriteAsync("bob", Team, u.Subject, u.Kind, u.NewKey, u.New, "payments");
newIds.Add(id);
wronglySuperseded.AddRange(olds.Where(o => nbIds.Contains(o)));
clock.Advance(TimeSpan.FromHours(1));
}
int correct = 0, stale = 0, wrong = 0;
for (var i = 0; i < Data.Updates.Length; i++)
{
var top = (await store.SearchAsync("dana", Data.Updates[i].Probe, k: 1)).FirstOrDefault();
if (top?.Id == newIds[i]) correct++; else if (top?.Id == oldIds[i]) { stale++; Console.WriteLine($" STALE {Data.Updates[i].Probe} -> {top!.Value}"); } else { wrong++; Console.WriteLine($" WRONG {Data.Updates[i].Probe} -> {top?.Value}"); }
}
var nbCorrect = 0;
for (var i = 0; i < Data.Neighbours.Length; i++)
if ((await store.SearchAsync("dana", Data.Neighbours[i].Probe, k: 1)).FirstOrDefault()?.Id == nbIds[i]) nbCorrect++;
Console.WriteLine($"[stale] {label,-12} updates correct={correct} stale={stale} wrong={wrong} | neighbours {nbCorrect}/{Data.Neighbours.Length} | wrongly superseded={wronglySuperseded.Count}");
if (label == "key")
{
await store.WriteAsync("bob", Team, "service:billing", Episode, "freeze", "Deploy freeze on billing until Friday", "payments", TimeSpan.FromHours(72));
var before = (await store.SearchAsync("dana", "Is there a deploy freeze on billing right now?", 1))[0];
clock.Advance(TimeSpan.FromDays(4));
var after = (await store.SearchAsync("dana", "Is there a deploy freeze on billing right now?", 1)).FirstOrDefault();
Console.WriteLine($"[ttl] before: {before.Value} ({before.Score:F3}) | after 4 days: {after?.Value} ({after?.Score:F3})");
}
}
// ---------------------------------------------------------------- permission-leak scenario
foreach (var (label, opt) in new[]
{
("v1-naive", new StoreOptions(ScopeAwareSupersession: false, PreFilter: false, ConsolidatePerScope: false)),
("v2-fixed", new StoreOptions()),
})
{
var (store, clock, db) = await Fresh($"leak_{label}", opt);
await Data.SeedLeakScenario(store, clock);
await store.ConsolidateAsync("service:search", "search");
await store.ConsolidateAsync("service:billing", "payments");
var privateTexts = await db.Entries.Where(e => e.Scope == Private).Select(e => e.Value).ToListAsync();
int leaked = 0, underfilled = 0;
foreach (var (user, q) in Data.LeakProbes)
{
var hits = await store.SearchAsync(user, q, k: 3);
if (hits.Count < 3) underfilled++;
var ownPrivate = await store.Visible(user).Where(e => e.Scope == Private).Select(e => e.Value).ToListAsync();
var forbidden = privateTexts.Except(ownPrivate).ToList();
foreach (var h in hits)
{
var mayReadRow = await store.Visible(user).AnyAsync(e => e.Id == h.Id);
var carriesPrivate = forbidden.Any(p => h.Value.Contains(p)); // e.g. a team summary built from someone's private note
if (!mayReadRow || carriesPrivate) { leaked++; Console.WriteLine($" LEAK {user}: {q} -> {h.Value[..Math.Min(90, h.Value.Length)]}"); }
}
}
var teamFact = await db.Entries.SingleAsync(e => e.Scope == Team && e.Key == "deploy.window");
var chain = label == "v1-naive"
? await NaiveCurrent(db, teamFact.Id)
: await store.CurrentVersionAsync("bob", teamFact.Id);
Console.WriteLine($"[leak] {label,-9} leaked hits={leaked} underfilled={underfilled}/{Data.LeakProbes.Length} team fact superseded by private note={teamFact.ValidUntil is not null} history shows: {chain}");
if (label == "v2-fixed")
{
var needles = new[] { "interviewing for the open backend role", "async standups", "Alice" };
var (n, v) = await store.DeleteSubjectAsync("user:alice");
var proof = await store.ProveGoneAsync("user:alice", needles, "runs/leak_v2-fixed.db");
Console.WriteLine($"[erase] deleted entries={n} vectors={v} rowsLeft={proof.RowsLeft} orphanVectors={proof.OrphanVectors}");
Console.WriteLine($" live rows still mentioning a needle: [{string.Join(",", proof.LiveMentions)}]");
Console.WriteLine($" nearest remaining vector per needle: [{string.Join(", ", proof.NearestCosine.Select(x => x.ToString("F3")))}]");
Console.WriteLine($" needles found in raw file bytes: {string.Join("; ", proof.BytesOnDisk.Select(kv => $"{kv.Key}=[{string.Join(",", kv.Value)}]"))}");
}
}
// ---------------------------------------------------------------- helpers
async Task<(MemoryStore, ManualClock, MemoryDb)> Fresh(string name, StoreOptions opt, bool secureDelete = true)
{
var path = $"runs/{name}.db";
foreach (var s in new[] { "", "-wal", "-shm", "-journal" }) if (File.Exists(path + s)) File.Delete(path + s);
var options = new DbContextOptionsBuilder<MemoryDb>()
.UseSqlite($"Data Source={path};Pooling=False")
.AddInterceptors(new SqlitePragmas(secureDelete))
.Options;
var db = new MemoryDb(options);
await db.Database.EnsureCreatedAsync();
await db.Database.ExecuteSqlRawAsync("PRAGMA journal_mode=WAL;");
db.TeamMembers.AddRange(
new TeamMember { UserId = "alice", TeamId = "payments" }, new TeamMember { UserId = "bob", TeamId = "payments" },
new TeamMember { UserId = "dana", TeamId = "payments" }, new TeamMember { UserId = "carol", TeamId = "search" });
await db.SaveChangesAsync();
var clock = new ManualClock(new DateTimeOffset(2026, 10, 1, 9, 0, 0, TimeSpan.Zero));
return (new MemoryStore(db, embedder, clock, opt), clock, db);
}
// what my first history endpoint did: follow the chain, no permission check
static async Task<string> NaiveCurrent(MemoryDb db, long id)
{
while (true)
{
var e = await db.Entries.SingleAsync(x => x.Id == id);
if (e.SupersededById is null) return e.Value;
id = e.SupersededById.Value;
}
}
public sealed class ManualClock(DateTimeOffset start) : TimeProvider
{
private DateTimeOffset _now = start;
public override DateTimeOffset GetUtcNow() => _now;
public void Advance(TimeSpan by) => _now += by;
}
public static class Data
{
public record Update(string Subject, MemoryKind Kind, string Key, string Old, string NewKey, string New, string Probe);
public record Neighbour(string Subject, MemoryKind Kind, string Key, string Value, string Probe);
public static readonly Update[] Updates =
[
new("service:staging", Fact, "db.engine", "Staging runs on Postgres 14", "db.engine", "Staging runs on Postgres 16", "What Postgres version does staging run?"),
new("service:billing", Fact, "deploy.window", "Billing deploys only on Tuesdays and Thursdays", "deploy.window", "Billing deploys any weekday before 3pm", "When is billing allowed to deploy?"),
new("service:billing", Decision, "retry.policy", "Payment retries use three attempts with a fixed 5 second delay", "retry.policy", "Payment retries use exponential backoff capped at five attempts", "How do payment retries work?"),
new("team:payments", Fact, "oncall.owner", "Bob owns the payments on-call rotation this quarter", "oncall.owner", "Dana took over the payments on-call rotation", "Who is on call for payments?"),
new("service:search", Fact, "index.engine", "Search uses Elasticsearch 7 for indexing", "index.engine", "Search migrated its index to OpenSearch 2", "What engine does search indexing use?"),
new("service:billing", Fact, "currency.default", "Default invoice currency is USD", "currency.default", "Default invoice currency is EUR", "What currency are invoices issued in?"),
// same fact, new key: the writer did not know the old key existed
new("service:auth", Fact, "token.lifetime", "Access tokens live for 60 minutes", "session.ttl", "We cut access token lifetime to 15 minutes after the audit", "How long do access tokens last?"),
new("service:billing", Fact, "queue.broker", "Billing events go through RabbitMQ", "messaging", "Billing moved its event stream to Kafka", "What broker carries billing events?"),
new("service:search", Fact, "owner", "Carol's team owns the search ranking service", "maintainers", "Ranking service ownership moved to the platform team", "Who owns the search ranking service?"),
new("team:payments", Preference, "review.style", "Payments team wants pull requests under 400 lines", "pr.size", "Payments now asks for pull requests under 200 lines", "How big can a payments pull request be?"),
];
public static readonly Neighbour[] Neighbours =
[
new("service:staging", Fact, "replica.db", "Staging read replica runs on Postgres 14", "What does the staging read replica run on?"),
new("service:staging", Fact, "cache.engine", "Staging cache runs on Redis 7", "What cache does staging use?"),
new("service:auth", Fact, "refresh.lifetime", "Refresh tokens live for 30 days", "How long do refresh tokens last?"),
new("service:billing", Fact, "currency.reporting", "Finance reports are produced in USD", "What currency are finance reports in?"),
new("service:billing", Fact, "refund.window", "Refunds are allowed within 30 days", "How long is the refund window?"),
];
public static readonly (string User, string Query)[] LeakProbes =
[
("bob", "When is billing allowed to deploy?"),
("bob", "What does Alice prefer for standups?"),
("carol", "Who is interviewing for the backend role on the search team?"),
("carol", "Is search being reorganised or merged?"),
("carol", "What changes are coming to search in Q1?"),
("dana", "Who owns the billing retry runbook?"),
("carol", "What is Alice working on?"),
];
public static async Task SeedLeakScenario(MemoryStore s, ManualClock clock)
{
await s.WriteAsync("bob", Team, "service:billing", Fact, "deploy.window", "Billing deploys any weekday before 3pm", "payments");
await s.WriteAsync("bob", Team, "service:billing", Fact, "runbook.owner", "Alice owns the billing retry runbook", "payments");
await s.WriteAsync("carol", Team, "service:search", Fact, "index.engine", "Search migrated its index to OpenSearch 2", "search");
await s.WriteAsync("carol", Team, "service:search", Decision, "ranking", "Search ranking moves to a learned model in Q1", "search");
await s.WriteAsync("carol", Team, "team:search", Fact, "hiring", "The search team has one open backend role", "search");
clock.Advance(TimeSpan.FromHours(2));
await s.WriteAsync("alice", Private, "user:alice", Fact, "career", "Alice is interviewing for the open backend role on the search team");
await s.WriteAsync("alice", Private, "user:alice", Preference, "standup", "Alice prefers async standups and no meetings before 10am");
await s.WriteAsync("alice", Private, "service:billing", Fact, "deploy.window", "My own note: I only deploy billing on Mondays because of the finance close");
await s.WriteAsync("alice", Private, "service:search", Fact, "reorg", "Heard from my manager that search will be merged into platform in Q1");
clock.Advance(TimeSpan.FromHours(2));
}
}
How I actually ran it, and what that means for the numbers
I have to be straight about this part. The machine I did this on could install the .NET 10 SDK but couldn’t reach NuGet or the Ollama model registry. So the C# above compiles (I checked it against stand-ins for the EF Core surface it uses), but I could not execute the EF queries or call nomic-embed-text there.
To still get real numbers instead of guesses, I ran the same store as a line-for-line Python port: same schema, same SQL shape, same supersession, TTL, filter, consolidation and erasure rules, against real SQLite files. For embeddings it used WordLlama’s l2_supercat model, a 256-dimension static embedding model that ships inside its pip package, so it runs offline.
What carries over and what doesn’t:
- The permission and erasure results carry over. They come from SQL and from SQLite’s file format, not from the embedding model. EF Core uses the same SQLite underneath.
- The stale-fact retrieval numbers do not fully carry over. A static word-embedding model is much weaker than nomic-embed-text at matching a question to an answer. Expect fewer "wrong retrieval" results with Ollama. I don't expect the supersession findings to change, for reasons you'll see.
I’ve kept the port and its raw output next to the C# project, so the numbers can be checked.
Scenario 1: stale facts
The setup is a payments team’s memory. Alice writes ten facts. Two weeks later Bob updates all ten. Six of his updates reuse the original key (db.engine stays db.engine). Four use a new key for the same fact, because Bob didn't know the old key existed: token.lifetime becomes session.ttl, queue.broker becomes messaging. That second group is the realistic one. Nobody looks up the existing key before telling an agent something changed.
I also added five neighbour facts that look like the updated ones but are different facts, like “Staging read replica runs on Postgres 14” sitting next to “Staging runs on Postgres 14”. Nothing should ever replace those.
Then Dana, a teammate, asks one question per fact, and I check the top result.
Stale-fact scenario: top-1 answer for Dana (10 updated facts, 5 neighbours)
variant updated facts same-key (6) new-key (4) neighbours
correct stale wrong c / s / w c / s / w intact
---------------------- ------- ----- ----- ------------ ----------- ----------
raw (no supersession) 4 5 1 2 / 3 / 1 2 / 2 / 0 5 / 5
key supersession 7 2 1 5 / 0 / 1 2 / 2 / 0 5 / 5
key + cosine >= 0.90 7 2 1 5 / 0 / 1 2 / 2 / 0 5 / 5
key + cosine >= 0.80 7 2 1 5 / 0 / 1 2 / 2 / 0 4 / 5
key + cosine >= 0.70 7 2 1 5 / 0 / 1 2 / 2 / 0 4 / 5
What worked
Key supersession did exactly its job. Every same-key update that was retrieved came back as the current value. The stale answers in the raw store (“Payment retries use three attempts with a fixed 5 second delay”, “Bob owns the payments on-call rotation”, “Search uses Elasticsearch 7”) are exactly the pricing-plan bug from Sakshi’s story, and closing the old row with ValidUntil removed all three.
TTL worked too. Before expiry, “Is there a deploy freeze on billing right now?” returned the freeze entry with a cosine of 0.73. Four days later the freeze was gone from results.
What failed
Failure 1: supersession only works when the writer reuses the key. Two of the four new-key updates still came back stale: “Access tokens live for 60 minutes” beat the newer 15-minute rule, and “Carol’s team owns the search ranking service” beat the ownership change. Key supersession did nothing for them, because there was no key match. The other two new-key updates were “correct” only because the newer sentence happened to be closer to the question. That’s luck, not lifecycle.
Failure 2: similarity cannot tell “replaced” from “related”. The obvious fix is to supersede anything above a cosine threshold with the same subject and kind. Here are the similarities that decided it:
old fact -> new fact cosine should supersede?
----------------------------------------------------------------------- ------ -----------------
Staging runs on Postgres 14 -> Staging runs on Postgres 16 0.964 yes
Payments ... under 400 lines -> Payments now asks ... under 200 lines 0.861 yes
Staging read replica runs on Postgres 14 -> Staging runs on Postgres 16 0.868 NO, different fact
Access tokens live for 60 minutes -> We cut ... to 15 minutes ... 0.584 yes
Carol's team owns ... ranking -> Ranking service ownership moved ... 0.511 yes
Billing events go through RabbitMQ -> Billing moved its event stream ... 0.305 yes
The read replica fact, which is a different fact, is more similar to the new staging fact than four of the five real replacements are to their originals. At 0.80 the store silently closed the replica fact, and the two stale paraphrases still survived. There is no threshold in that list that catches the real replacements without killing the neighbour. A better embedding model will move these numbers, but I don’t think it changes the shape: similarity measures topic, and replacement is not a topic, it’s a relation. That is Sakshi’s point, and this is the first time I’ve seen it in my own data rather than agreeing with it in principle.
So I’m not shipping semantic supersession. What I’m doing instead is cheaper and more boring: when a write lands on a subject that already has live facts of the same kind, the agent gets the existing keys back and has to either reuse one or explicitly say “new fact”. That’s a submit_module-style gate on the write path, which is where Cekikj put his.
Failure 3: TTL expiry exposed a missing score floor. After the freeze expired, the top result for “Is there a deploy freeze on billing right now?” wasn’t empty. It was “Payments now asks for pull requests under 200 lines”, with a cosine of 0.17. The agent would have received an unrelated rule as its best evidence about a deploy freeze. The MinScore option in StoreOptions exists because of this run. I don't have a principled value for it yet; it depends on the model, and with nomic-embed-text I'll have to pick it from real query logs.
One wrong retrieval that isn’t a lifecycle bug. “When is billing allowed to deploy?” returned “Refunds are allowed within 30 days” in every variant. That’s the static model matching on the word “allowed”. I’m counting it as a model weakness and expect nomic-embed-text to fix it, but I haven't verified that.
Scenario 2: permission leaks
Same store, now with two teams. Alice, Bob and Dana are on payments. Carol is on search. There are five team facts and four private notes from Alice:
- “Alice is interviewing for the open backend role on the search team” (subject user:alice)
- “Alice prefers async standups and no meetings before 10am” (subject user:alice)
- “My own note: I only deploy billing on Mondays because of the finance close” (subject service:billing, key deploy.window, the same key as a team fact)
- “Heard from my manager that search will be merged into platform in Q1” (subject service:search)
Then the nightly consolidation job runs and builds a summary per service for the team. Seven probes from Bob, Carol and Dana, top-3 each. A leak is any hit the user can’t see, or any visible hit that carries someone else’s private text.
My v1 is what I would have written a month ago. Supersession matched on subject, kind and key only. Retrieval ranked everything and filtered after. Consolidation summarised every live entry about a subject.
Permission scenario: 7 probes, top-3 each
variant leaked under-filled team fact killed history view
hits queries by private note shows private note
------------------------------------ ------ ------------ ---------------- ------------------
v1: scope-blind supersession, 5 7 / 7 yes yes
filter after ranking,
consolidate every scope
v1 with filter before ranking 7 3 / 7 yes yes
v2: scope-aware supersession, 0 0 / 7 no no
filter before ranking,
consolidate team rows only
What failed, in the order I found it
Leak 1: a private note superseded a team fact. Alice’s private “I only deploy billing on Mondays” had the same subject, kind and key as the team’s “Billing deploys any weekday before 3pm”. v1 supersession didn’t look at scope, so Alice’s note closed the team fact. Bob, asking when billing can deploy, lost the team answer. Nobody saw anything they shouldn’t have at that point. It was an availability bug, and I’d have filed it as “flaky retrieval”.
Leak 2: the history view walked straight into the private note. I had a “show current version” helper that follows SupersededById to the latest row. For Bob, starting from the team fact, it returned Alice's private note word for word. This is the leak that scares me, because history views are the kind of thing you add later, for debugging, without re-reading the permission model. The fix in CurrentVersionAsync is that every hop goes through Visible(userId) and an invisible hop stops the walk instead of skipping it.
Leak 3: every leaked search hit was a summary. This one surprised me. In both v1 variants, row-level permissions were correct: Carol never received Alice’s private row. What she received was the team-scoped summary for service:search, which the consolidation job had built from all live entries, including "Heard from my manager that search will be merged into platform in Q1". Carol asked "Is search being reorganised or merged?" and got Alice's manager's comment, filed under a scope Carol was entitled to read.
That’s the point where Atlassian’s “permissions at the data layer” rule stopped being a slogan for me. My filter was in the data layer. The leak came from derived data whose scope was assigned by a job, not inherited from its inputs. A summary has to carry the most restrictive scope of anything that went into it, or consolidate per scope. I chose per scope: private notes never feed a team summary. It’s also why the Codex consolidation_model setting matters more than it looks: consolidation is where scopes get mixed.
Filtering after ranking didn’t leak, it starved. With the filter after ranking, all seven queries came back with fewer than three results. “What is Alice working on?” from Carol came back empty, because the top three were all Alice’s private rows and got dropped. Moving the filter before ranking fixed the under-fill but raised the leak count from 5 to 7, because the freed slots got filled by the poisoned summaries. Fixing the filter alone made the measured leak worse. I wouldn’t have believed that without running it.
v2, with all three fixes, had no leaks, no under-filled queries, and the team fact survived.
Scenario 3: delete by subject, and proving it
Alice leaves and asks for her data to be erased. DeleteSubjectAsync("user:alice") deletes her two user:alice entries and their two vectors. Then ProveGoneAsync checks five things: rows left for the subject, orphaned vectors, live rows still mentioning her, the nearest remaining vector to the deleted text, and whether the deleted text is still present in the raw database file bytes.
I ran it four ways: SQLite’s secure_delete off and on, with the default rollback journal and with WAL.
Erasure of subject "user:alice": 2 entries, 2 vectors deleted
check secure_delete OFF secure_delete ON
----------------------------------------- ------------------------- -------------------------
rows left for subject 0 0
orphaned vectors 0 0
deleted text in file after DELETE yes no
deleted vector blobs in file after DELETE 2 of 2 0 of 2
after VACUUM (rollback journal) clean clean
after VACUUM (WAL, no checkpoint) text still in main file clean
after VACUUM + wal_checkpoint(TRUNCATE) clean clean
live rows still mentioning "Alice" 2 2
What failed
Failure 1: the SQL proof passed while the data was still on disk. With secure_delete off, which is SQLite's usual default, the row count was 0, the orphan count was 0, and both the private text and both raw vector blobs were still sitting in free pages of the database file. Any proof that only runs queries would have declared victory. PRAGMA secure_delete=ON zeroes the freed content at delete time. Note that it's a per-connection setting, which is why the C# applies it in a connection interceptor rather than once at startup.
Failure 2: VACUUM in WAL mode isn’t enough by itself. In WAL mode, the rewritten pages from VACUUM go into the -wal file. Until the next checkpoint, the main database file still contains the old pages, deleted text included. Only VACUUM followed by PRAGMA wal_checkpoint(TRUNCATE) gave a clean file. I didn't know this before running the test, and I'd have shipped "we run VACUUM after erasure" as the answer.
Failure 3: delete by subject deletes a subject, not a person. Two live rows still mention Alice: the team fact “Alice owns the billing retry runbook” and the billing summary built from it. And her private note about the search reorg survived untouched, because I filed it under service:search, not user:alice. Both of those are correct behaviour for "delete subject user:alice" and both are wrong for "erase Alice". Erasure needs at least three passes: by subject, by owner, and a mention scan whose results a human reviews, because "Alice owns the runbook" is a team fact that may legitimately need to be rewritten rather than deleted. This is Wasowski's point arriving in my own output.
The nearest-neighbour check is not a proof either. The remaining vector closest to “interviewing for the open backend role” had a cosine of 0.66. It belongs to the team fact “The search team has one open backend role”, which is a legitimate neighbour, not residue. A similarity check can tell you something semantically close remains. It can’t tell you whether that’s a leftover or a different fact. I keep it in the proof as a pointer for review, not as a pass or fail.
What the proof doesn’t cover. Backups. Logs. Anything the embedding provider retained (local Ollama helps here). And my own OllamaEmbedder cache, which keeps every embedded string in a dictionary for the life of the process. In a long-running service that cache is a copy of the memory store that no erasure path touches. I left it in the code above on purpose so you can see it; in my real service it's going to be keyed by hash and cleared on erasure.
Approach by failure mode and cost
This is my summary of the approaches from this week, my own runs included, by what they catch, what they miss, and what they cost. The cost column is my judgement from building or reading about each, not a measurement.
approach catches misses cost
------------------------------- ------------------------------- ---------------------------------- -----------------------------
raw vector store nothing about time or scope stale facts, leaks, erasure lowest to build, highest to
debug later
key supersession same-key updates paraphrased updates under a new one indexed query per write
(ValidUntil + SupersededById) key (2 of 4 in my run)
cosine supersession some paraphrased updates replaced distinct neighbours at one vector scan per write,
0.80; missed 2 real updates silent data loss
valid_from / valid_until edges "what was true when" questions only as good as the code that a graph store, or two columns
(Sakshi's framing) writes the edges and discipline
write gate (submit_module unsupported or duplicate judge errors; same-vendor judge one model call per write
style) claims before they persist shares generator blind spots
TTL on episodes short-lived facts garbage top hit after expiry trivial; needs a score floor
without a score floor
permission filter after nothing leaks from rows starves results (7 of 7 queries cheap, and wrong
ranking under-filled in my run)
permission filter before row-level leaks, under-fill leaks via derived data (7 leaked one join per query
ranking (SQL) hits from summaries in my run)
scope-inheriting consolidation private text in team summaries cross-scope insight is lost more summaries, one per scope
delete by subject rows + vectors for one subject mentions elsewhere, owner-filed one transaction
notes, caches, backups
+ secure_delete + VACUUM + deleted bytes left on disk backups, provider-side retention write amplification on delete,
wal_checkpoint(TRUNCATE) full-file rewrite on VACUUM
managed graph (Teamwork Graph linking, scopes, audit, only what is in the vendor's vendor licence; 44% / 48% is
style) permissions at source graph; method not published an unpublished internal number
learning loop with receipts traceable behaviour changes improvement claims, if the eval GPU or rule-writing time; a
(Reef style) is only an echo health check real eval is on you
What I’m changing, and what I still don’t know
What I’m changing in my own agents:
- Supersession stays key-based. Writes to a subject with existing live facts of the same kind get the existing keys back, and the agent has to reuse one or declare a new fact. No similarity-based replacement.
- Every derived row (summaries, consolidations, anything a job writes) inherits the most restrictive scope of its inputs. Consolidation runs per scope.
- Every read path, including debug and history views, goes through one Visible(userId) query. A history walk stops at the first invisible hop.
- secure_delete=ON on every connection, and erasure ends with VACUUM plus wal_checkpoint(TRUNCATE).
- Erasure runs by subject, by owner, and as a mention scan that produces a review list.
- A minimum score on retrieval, tuned from real logs.
What I still don’t know:
- How much of the stale-fact table changes with nomic-embed-text. I expect fewer wrong retrievals. I don't expect cosine supersession to become safe, but that is a prediction, not a result.
- Whether my Visible() query translates the way I think in EF Core 10. The first thing I'll do on a real box is print it with ToQueryString() and read the SQL.
- What the Teamwork Graph benchmark measured. If Atlassian publishes the method, I’d like to see the baseline.
- Whether Reef’s loop improves anything with a real evaluation plugged in. The receipt header is the part I’d steal today.
- How to handle backups for erasure without restoring and rewriting them. Every approach I know of is either expensive or a policy promise.
The common thread this week was that memory is a database, and the boring database questions (who can read this row, what replaced it, is it really gone from the file) are where my bugs were. Apart from one retrieval miss I’m blaming on a weak embedding model, none of my failures were about retrieval quality. They were about lifecycle and scope.
Sources
- Atlassian, Teamwork Graph at Team ’26 (vendor; source of the 44% and 48% figures): https://www.atlassian.com/blog/company-news/teamwork-graph-team-26
- Atlassian Community, Teamwork Graph CLI GA (vendor; scopes, audit logs, “internal study”): https://community.atlassian.com/forums/Atlassian-AI-Rovo-articles/Teamwork-Graph-CLI-GA-connected-context-at-enterprise-scale/ba-p/3254146
- The Code newsletter, 15 October 2026 (secondhand summary of the Teamwork Graph rules and the Codex memory tip)
- Jaroslaw Wasowski, “The Right to Be Forgotten in AI Agent Memory”: https://levelup.gitconnected.com/the-right-to-be-forgotten-in-ai-agent-memory-gdpr-garbage-collection-and-contradictions-e6b39e2e33c4
- Miodrag Cekikj, course compiler (author-reported, indicative, same-vendor judge): https://medium.com/ai-advances/we-dont-learn-the-same-way-now-the-course-can-adapt-without-making-things-up-5538c5501410
- Kristopher Dunham, Reef (project-reported numbers): https://medium.com/@creativeaininja/your-ai-agent-doesnt-learn-from-its-mistakes-reef-is-building-the-missing-loop-b41342cf2dfa
- Reef repository: https://github.com/Human-Agent-Society/reef
- Gowtham Boyina, AutoResearch write-up: https://medium.com/towards-artificial-intelligence/what-karpathys-630-line-script-does-at-amazon-scale-4f9ed20d73b0
- Chandran, Kim, Jha, Castells, Hottier, “AutoResearch at Production Scale: Failure Modes and a Multi-Agent Framework” (authors’ reported results): https://arxiv.org/abs/2609.30541
- OpenAI Codex source, memory config keys: https://github.com/openai/codex/blob/main/codex-rs/config/src/types.rs
- Sakshi, graph vs vector memory (benchmark numbers not reused): https://medium.com/data-science-collective/three-questions-that-tell-you-in-90-secondswhether-you-need-graph-memory-or-vector-memory-ebd7ce8bc1d3
- Sakshi, 5 .md files every AI agent needs (unmeasured claims): https://ai.plainenglish.io/5-md-files-every-ai-agent-needs-in-its-repo-right-now-29eea89c830f
- SQLite documentation, secure_delete pragma: https://www.sqlite.org/pragma.html#pragma_secure_delete
- My earlier piece, “Your Agent’s Memory Is Probably Wrong”: https://topuzas.medium.com/your-agents-memory-is-probably-wrong-a-practical-guide-to-agentic-memory-design-0b3fb0915612
Tags: AI Agents, Agent Memory, LLM Engineering, DotNet, Entity Framework Core, SQLite, Ollama, Data Privacy, GDPR, Retrieval Augmented Generation
Top comments (0)