Building in public
You explain your deploy setup to your assistant. It helps. Tomorrow you explain the same setup again. And the day after. You ar...
For further actions, you may consider blocking this person and/or reporting abuse
Counting the saves is the part most people skip. After a few weeks of running agents with any memory layer, the useful metric is not whether it recalled something, but how many times recall stopped a repeated deploy mistake. The silent field drop is worse than an empty result, because the agent keeps going with half a fact and sounds confident about it. Teaching the tool to say why nothing came back is half the product. Empty silence reads as broken to anyone who did not write the server.
Great write-up - the HIT/MISS ratio is a much cleaner instrument than the "did it feel useful" vibes most of us run on. One data point from a different corner of the same problem: instead of a separate recall store, I've been keeping the durable memory inside the repo the assistant works on - every fix commits with the trigger (the user complaint) and the rationale in the message, so "why did we do this" is a
git logaway and versioning/rollback come free. The trade-off I hit: retrieval is grep/diffs, not semantic - strong for "what did we decide about X", weak for "what did we learn that's similar-but-not-the-same as X". The MCP store has the opposite strength.Question on the versioning side: when a saved lesson gets superseded (you changed the deployment and the old note is now actively wrong), how does cachly handle that? Can you see what changed and why, or does it just overwrite? That's the one thing a git-based approach gives you for free that I'd miss in a plain save/recall store.
Your trade-off is the one I would have written down too, and you named it more precisely than I did in the post.
On the versioning question - three parts, because two have a clean answer and one does not.
Does it overwrite? For the answer, yes. A lesson is keyed by topic, so writing under the same topic replaces the current text. That is deliberate: a superseded note that keeps competing on relevance is worse than no note at all.
Is the old one gone? No. Every write also appends the complete record to a per-topic history list. I counted in my own store rather than guessing: 524 lessons, of which 257 (49%) have been overwritten at least once - 408 overwrites in total. 638 superseded versions are still sitting there, across 334 topics. So the "what changed" half of your question has a real answer: the previous texts are there and they diff.
Can you see why? No. That is the honest gap, and it is exactly the half your approach gets for free. Each record carries an audit entry with a timestamp, whether it was a create or an update, and the previous outcome. There is no rationale field. Your commit message has the trigger and the reasoning in it; my update has neither.
Three more things I would rather say than have you discover:
One design decision that is in there and belongs in the same family: a record whose outcome is a failure cannot take the slot from a known-good fix. Success and partial always replace; a failure only claims the slot when nothing is there. Otherwise one bad run overwrites the fix with the report that it broke.
On not having to pick a side - we ended up reading your half into ours rather than choosing.
brain_from_gitparses commit history and turns fixes into lessons, incrementally, so re-running only picks up new commits. And there are two CI templates, GitHub Actions and GitLab CI, that write a lesson per run, so a red-to-green transition gets recorded as a learned fix. The reasoning was the one you describe: the commit message already carries the trigger and the rationale, so not reading it means throwing away the best-labelled data in the repo.What that still does not do is carry the rationale through into the lesson as a first-class field. It reads your history; it does not keep your "why". That is the piece your approach has and mine does not, and it is now written down as a gap rather than a preference.
That's the honest answer I was hoping for, especially the parts you'd rather say than have someone discover.
The 90-day expiry is the one I'd push back on. Your own counts show most corrections land within days, but the cases that actually hurt are the ones that take quarters to surface - and "the version that was right is already gone" is exactly when the old text becomes the most valuable record you own. If it's a storage concern, keeping only superseded versions (not every write) gets most of the safety at a fraction of the space.
On the missing rationale - a cheap discipline that worked for us: require a one-line "why" on every update, same as a commit message. The diff says what changed; the one-liner says why it was wrong then. Two months later that's the line that gets read. You already diff the per-topic history, so it's one more field on the update record - and it feeds the "why did this keep changing" view that plain diffs never show.
Both points land, and the first one lands harder than you may know.
On the 90 days: your sharpening is exactly right, and I can now back it with something that happened here yesterday, not two quarters ago. We removed a retrieval feature whose justifying measurement was less than 24 hours old - and already unreproducible on the frozen benchmark, because the pipeline around it had moved. The question "which version was right, and when" turned out to be the load-bearing one on a one-day-old fact. An expiry that deletes superseded versions is deleting precisely the records whose value peaks late.
Your compromise is the right shape: keep superseded versions only, not every write. Most of the safety, a fraction of the space. That is now the stated target on our board, with your name on the card: superseded versions become exempt from the TTL; the expiry stays for everything else.
On the one-line why: what convinced me is not the discipline argument - it is that we already believe it and act inconsistently. Our git importer exists because commit messages carry the trigger and the rationale; that was the whole pitch. Then our own update path throws exactly that information away. We justified the import with a field we do not keep ourselves.
One thing makes "require it" cheaper for us than for most tools: the writer is almost always a model mid-session. It knows, in that moment, why it is updating - the field costs a human nothing and the model half a sentence. Required fields filled by humans rot into "fix"; required fields filled by an assistant that just diagnosed the problem tend to contain the diagnosis. So: required on update, free-form on first write, and the why travels into the superseded version's history - which is the point where your two suggestions turn out to be one feature. A kept old version without its why is trivia; a why without the version it explains is a slogan. Together they are the git log we claimed to envy.
The under-24-hours example is the strongest possible confirmation - it collapses the "late" failure into a window you actually measure. And the "which version was right, and when" framing is exactly the question an audit log answers that a current-state store can't.
The required-on-update point is the one I'll steal. I'd reflexively assumed "required = humans rot it into 'fix'" and never considered that the writer being a model mid-session changes the economics - it genuinely has the diagnosis in context at write time, so the field costs it half a sentence. That's the same reason our commit messages carry the trigger: the assistant that just hit the problem is the one writing the rationale, and it's the only writer who knows what the rationale is.
One connection that follows from "the why travels into the superseded version's history": it turns your 638 superseded versions from a dump into a navigable topic-log. You already admitted the missing diff-surfacing UI - once each superseded version carries its why, the "show me the history of this topic" view writes itself, and the diff becomes the story rather than a forensic chore. The data was always there; the why is the index that makes it browsable.
"The why is the index that makes it browsable" is the sentence that reorganised
this for me. I had the two as separate cards - keep superseded versions, add a
rationale field - and you just showed they are one feature with a UI falling
out of it for free. A kept version without its why is trivia; a why without its
version is a slogan.
One thing I found today that belongs next to your TTL argument, because it is
worse than the case we were discussing.
The 90-day expiry does not only sit on the history. It also sits on the
dependency index - the structure that lets a record say "I am true until that
config changes". So the one mechanism we have for invalidating a fact by
event is itself invalidated by age. Ninety days after the link is written it
is gone, and the fact it was guarding goes back to looking permanently true.
That is the same axis error you called out, one layer down, and I only saw it
because another commenter in this thread pushed on the write path and sent me
into the code. Your "superseded versions exempt from the TTL" now reads to me
as a narrower version of a rule I should state once: anything whose job is to
mark something else as possibly-wrong must not expire on a timer. The history
and the dependency index are both that.
Card updated with both, and your name is still on it.
I like the one-user phase here because it gives you a clean failure log. The next metric I’d want is harsher than recall count, though. How often did the memory change the action the assistant took, and how often did you have to correct that memory afterward? That separates useful context from confident clutter.
Correction to my own comment above/below, and it is the worst kind of one.
I wrote: "Retrieval quality itself is benchmarked and defended in CI - +33% Precision@1 over raw BM25, 98.2% Recall@3, reproducible with one command."
I ran the command again. This is what it prints:
Not +33%. +0.0% - and ten points behind a flat file on the first metric.
How the wrong number survived: it was a hardcoded string in the server's own metrics output, with a comment above it reading "CI-defended", and a unit test asserting that the string was present. So there was a green check. The check was guarding the sentence, not the measurement. A number the server does not compute cannot fall, which means it is not measuring anything.
The instrument is worse than the string, and that matters more. That benchmark has 17 lessons and 13 queries. On 498 real lessons with 20 questions asked in plain language, the current ranking formula scores 30% Precision@1 and the one it replaced scores 15% - while the 17-lesson benchmark ranks them the other way round, 69% against 92%. On the one comparison I can check against reality, it moves in the opposite direction.
So the honest state: the only retrieval number I can stand behind is 30% Precision@1 on 498 real lessons, up from 15%, measured once, on one store, by me. Everything I said about CI-defended benchmarking was describing a test that defended a sentence.
The string is out of the product, and the test now asserts the opposite - that no fixed ranking number appears in that output at all. Replacing the benchmark with something larger than 17 lessons is the next job, and it has to happen before any number goes back in.
Your question was whether the memory changed the action. I answered it and then attached a caveat that turned out to be the less honest half of the comment. The metric you asked for is what sent me to run the command, so: thank you, twice over.
"Confident clutter" goes into my vocabulary, with attribution.
Your second question sent me to count instead of guess, and the counting corrected me twice.
First pass: I searched all 521 records for ones that explicitly correct an earlier record - phrases like "corrects a previously stored lesson". 48 of them. I was about to write that number down as a defect.
Second pass, splitting them properly: 39 of the 48 correct their own record. The store updates in place and bumps a version, so the wrong text is gone rather than sitting next to its replacement. Of the remaining 9, only 2 name another record that still exists - and reading those two, they are citing examples, not superseding anything.
So the honest answer to "how often did you have to correct the memory": roughly one record in ten has been corrected at some point, and the correction almost always replaced the thing it corrected rather than competing with it. That is a considerably better answer than the one I was about to give, and I only have it because you asked for a metric instead of an impression.
What is genuinely missing is narrower than I first thought: a record cannot say "I supersede that one" across topics, and there is no valid-from date, so retrieval cannot prefer the currently applicable fact over an older one that still looks true. I have not yet found a case where that bit us. I would rather say that than dress up a gap I cannot demonstrate.
Your first question is the harder one, and you phrased it better than I had. "Did it change the action" is not "was it used". Used is self-reported and will flatter. Changed-the-action is a counterfactual, and counterfactuals do not come from asking.
The only honest route I see is to withhold. Hold back the top record on a random half of eligible turns; if the outcomes differ, the record changed the action. Not per turn - in aggregate, which is the level the claim gets made at anyway.
That is what I am setting up. The awkward part belongs in the same breath: it costs real sessions, because half my own turns get a deliberately worse answer for the duration. That is the price of the number, and I have not found a cheaper one that is not a self-report in disguise.
One thing I should put next to all this, because otherwise it reads as if nothing here is measured. Retrieval quality itself is benchmarked and defended in CI — +33% Precision@1 over raw BM25, 98.2% Recall@3 against an external corpus, reproducible with one command. What I cannot yet prove is the step after that: whether finding the right record changes what the assistant does. That is the gap the hold-out is for, and it is the honest boundary between what I can show you and what I am still owed.
Counting prevention instead of usage is the reframe I didn't know I needed. I ran into something similar with Steering documents in Kiro during a hackathon, where the value only showed up as an absence, bugs that quietly never happened because the context was already there. The team memory angle in cachly is the problem I haven't figured out yet, checking it out!
"The value only showed up as an absence" is the whole measurement problem in one line, and you got there from a different direction than I did.
That's why the metric is so slippery: an event that didn't happen leaves no row in any table. You can count recalls, you can count writes - you cannot count the bug that never got filed. Every number I have measures delivery and I keep having to stop myself from labelling it prevention.
The only honest instrument I've found is withholding. Hold back the top record on a random half of eligible turns and compare outcomes in aggregate. It costs real sessions - half your own turns get a deliberately worse answer - and I haven't found a cheaper version that isn't a self-report in disguise. If Kiro's steering docs gave you any way to see the absence directly, I'd genuinely like to hear it.
On team memory, the honest state: it's built and it runs - every lesson carries the name of whoever learned it, and there's cross-author reuse tracking, which is the number that matters (how often you recall something a teammate wrote). What I don't have is teams using it. So the feature exists and the evidence doesn't.
The failure mode I expect first, and I'd rather name it before you find it: a lesson written in one person's vocabulary that nobody else ever retrieves. It counts as "stored" and prevents nothing. If you try it with a team, that's the number I'd watch - the share of lessons never recalled by anyone but their author.
The honest answer to your question is that Kiro's steering docs don't expose the causal mechanism directly either. What we can show is the document itself: explicit rules covering no comments in any form, strict TypeScript with no any, one responsibility per function with a thirty line limit, strict layer separation, and even commit message format and style, all applied without a single observed violation across hundreds of generated files in manual review.
The claim we can make with confidence is that the generated code was consistent with those rules from start to finish. What we cannot claim is that Steering corrected anything in the moment, because we never captured a case where Kiro generated a comment and then suppressed it. The absence of violations is observable. Whether there were generation attempts that got corrected is not visible to us.
Your withholding idea is the only method I've seen that would actually get at the causal question. The cost you describe is real and I haven't thought of a cheaper version either.
On the vocabulary problem, that's the failure mode I'd be most worried about too. A lesson stored in one person's framing that nobody else retrieves doesn't prevent anything. If I get to test cachly with a team I'll watch that number first
"The absence of violations is observable. Whether there were generation attempts that got corrected is not visible to us." - that sentence separates two things I've been running together all day, and I want to give you something back for it.
You don't need my expensive method. Your case is cheaper than mine, and it's cheaper for a specific reason: your rules are mechanically checkable.
No comments. No any. One responsibility, thirty lines. Layer separation. Commit message format. Every one of those is a thing a script can count in a file, without a human deciding anything.
So the experiment is not a withhold - it's an A/B on generation:
That works because you never have to observe the suppression. You only need the outcome under two conditions - and unlike my case, "was this output correct?" is a grep, not a judgement call.
Why my problem can't use that: my rule is "did the recalled lesson change what the assistant did", and there is no checker for that. The output isn't right or wrong in a countable sense; it's a different answer. That's the whole reason I ended up at withholding, and your case shows the boundary clearly: withholding is the price you pay for an unverifiable success criterion, not for a causal question. If your criterion is checkable, the cheap version exists.
One caveat before you run it, from being burned this weekend: run the checker on the without-steering set first, and confirm it actually reports violations. If it comes back clean on both, you haven't proved steering is unnecessary - you've proved your checker can't see. A guard that finds nothing and a guard that has nothing to find produce identical output.
On the vocabulary number: yes, that's the first one I'd watch too. If you do test with a team, the specific figure is share of lessons never retrieved by anyone but their author. High means the memory is measuring authorship.
That experiment design is exactly what I needed and I hadn't seen it because I was still thinking about it as an observation problem rather than a comparison problem. The checker-first step is the one I would have skipped and probably would have drawn the wrong conclusion from a clean result on both sides.
Going to run this properly: generate the same set of tasks with and without the steering doc, run a linter pass over both, and report the difference. The rules are specific enough that violations should be detectable if they exist. I'll share what comes out.
On the vocabulary number, share of lessons never retrieved by anyone but their author is the right metric. I'll track that if I get to test with a team.
One addition before you run it: the checker-first step is also the cheapest place to catch a broken experiment. If the linter finds zero violations in BOTH arms, suspect the checker before concluding the doc works - feed it one file with a deliberately planted violation first. That's the negative control's negative control. Looking forward to your numbers, and yes: share-of-lessons-never-retrieved-by-anyone-but-their-author is the metric I'd publish even solo - your future self counts as a second reader.
The implementation detail that matters most is making the assumption visible. For this kind of work I would put the invariant in CI or monitoring, then document the recovery path alongside it. That is how a one-time fix becomes a reliable operating practice.
Agreed on the principle, and I want to add the failure mode that sits right behind it - because "put the invariant in CI" is exactly what I did, and it went wrong twice in ways worth naming.
A guard can be green and blind. I had a check asserting that a benchmark number appeared in the server's output. The number was a hardcoded string. The check guarded the sentence, not the measurement - and a number the server does not compute cannot fall. It was green for weeks while the real figure was ten points behind a flat file.
A guard can watch the spelling instead of the rule. This one is from today. I wrote thirteen checks over a harvesting tool. Two hours later I rewrote the tool's transport - same rules, different words - and seven of thirteen went red, while nothing had gotten worse. The checks were asserting identifiers, not behaviour. I rewrote them to run the actual functions and only left a text assertion where an execution genuinely needs a network.
So the version of your rule I'd now write for myself: the invariant must be executed, not spelled. If a check can pass on a codebase where the thing it protects has been deleted and re-implemented differently, it's protecting the name.
Your recovery-path point is the half I'm weakest on and I'll take it plainly. My guards say "this is wrong"; most of them don't say what to do. The two I've fixed since read like this: the failing message names the replacement to use and prints the first twelve offending sites with file and line. That's the difference between a red check and an operating practice, and you named it better than I had.
That example makes the distinction concrete: a check can protect a string, an identifier, or the behavior we actually care about. The most useful guards exercise the boundary and then explain the recovery path in the failure output, so the next operator does not have to rediscover the rule under pressure.
The three-level ladder - a check can protect a string, an identifier, or the behavior - is worth keeping, because it doubles as an upgrade path: most guards are born on level one, and the honest question at review time is "which level is this, and is that the level we care about?" Your second point I'd underline twice: the failure output as the recovery manual. We've started writing guard messages as instructions to the next operator - not "count mismatch" but "this number may only go down; whoever lowers it records the new value HERE, that is the only allowed way to change this test." The rule travels with the failure, so the person under pressure at 2 a.m. gets the constitution, not just the verdict. A guard that explains its own recovery path is the difference between a tripwire and a colleague.
Great post. the "count the saves, not the usage" distinction is such a simple but sharp way to separate real tools from busywork.
The hit/miss log trick is a nice touch too, cheap way to get a real number instead of a gut feeling.
Question: how does cachly handle a saved lesson that turns out to be wrong later? Does it get overwritten, flagged, or just sit there until someone catches it?
Thank you - and that question has a precise answer, because I went and counted this afternoon instead of guessing.
Overwritten, mostly. A lesson is keyed by topic, so writing a correction under the same topic replaces the text and bumps a version; the audit trail keeps what changed. In my own store 488 of 521 records are past version 1, and of the 48 that explicitly say they correct something, 39 correct their own earlier text. So in the common case the wrong version is gone rather than sitting next to its replacement.
What does not happen, and this is the honest boundary:
A record cannot mark a different record as superseded. If the correction lands under a new topic name, both survive and compete on relevance alone.
And nothing detects rot. A lesson that quietly stopped being true - the server moved, the flag got renamed - sits there looking exactly as confident as the day it was written, until a human notices and writes the correction. There is no valid-from date, so retrieval cannot prefer "currently applicable" over "still plausible".
So the accurate answer to your three options: overwritten when someone catches it, and sitting there when nobody does. The first half is solid. The second half is the part I would rather name here than have a user discover on their own.
"Sitting there looking exactly as confident as the day it was written". that's the line. Most people asked this would've rounded up to "we handle it." You went and pulled the numbers instead.
The topic-keyed overwrite makes sense as a default. The real gap is no way to prefer "current" over "plausible", that's the honest next feature, not a nice-to-have.
Thank you - and since you were generous about the counting, I owe you a correction on one of the numbers I handed you.
I wrote "488 of 521 records are past version 1". I went back into the code today. That field is a schema version - a constant the writer stamps on every record to say which writer produced it. It is not a revision counter, and I read it as one.
The number I should have given you: 257 of 524 lessons (49%) have been overwritten at least once, 408 overwrites in total, 638 superseded versions still on disk. That happens to support the point more strongly than the wrong number did, which is not an excuse. I quoted a field without checking what it counted, one comment after asking someone else to stop trusting impressions.
On "prefer current over plausible" - you are right that it is the feature and not the polish, and it splits into two things that can be built separately.
The cheap half is a valid-from date, so retrieval can rank a currently-applicable fact above one that merely still looks true. The expensive half is supersession: letting one record point at another and say "this replaces it". The hard part there is not the link. It is noticing that the link should exist, when the correction lands under a different topic name and nothing mechanical connects the two.
Your framing exposed a third thing I had not separated out. Neither of those detects rot on its own. A lesson nobody ever contradicts, about a server that quietly moved, stays confident forever - there is no contradiction to find. Valid-from at least makes the age visible at ranking time instead of only in the record, which is the difference between "still plausible" and "still current" being a thing the ranker can see.
That is the roadmap now, in that order, and it came out of your comment rather than my planning.
Correcting a stat against your own point, unprompted, one comment after asking people to trust the numbers, that's the credibility move most people can't make.
The split you landed on is the right one: valid-from is cheap and ships now, supersession is the hard, real feature, and rot is the thing neither solves. Naming that gap yourself is worth more than shipping around it.
The one-user phase is underrated as a design phase. You get to change the schema on a Tuesday because you are the only person who would notice.
Reading the thread above on stale recalls, the part I would push on is writes rather than reads. Storing a fact is easy. Deciding what happens when a new one contradicts it is the whole product. Surfacing the contradiction, which you say cachly does, is the right instinct, but it hands resolution to whoever is reading. When that reader is an agent mid-task, it picks one and keeps going, and you find out later.
What worked for us on operational context was giving facts a reason to die instead of a confidence score. This is true until that config changes, and when the config changes the fact goes with it. Invalidation by event rather than by age or by vote.
Does cachly have any notion of a fact being scoped to something that can change underneath it, or is contradiction always resolved at read time?
You moved the question to the write path and that is where it belongs. I went
and read our own code rather than answer from memory, and the answer is worse and more interesting than a plain no.
Is there a notion of a fact scoped to something that can change underneath
it? Yes - and it does not reach the reader.
The write path takes a
depends_onlist (["node:>=20", "docker:running",) and builds a reverse index: dependency → the topics that"wireguard:active"]
rest on it. There is a
trace_dependencycall that walks that index and, withmark_review=true, stampsneeds_review: trueon every dependent record.That is exactly the shape you describe. Here is what is wrong with it.
One. I grepped the retrieval path for
needs_review. Zero occurrences. Notin the recall handler, not in the ranking core. The flag is written, and then
it is rendered as a badge inside
trace_dependency's own output - a toolnobody calls mid-task. A record marked
needs_reviewranks and returns exactlyas it did before. So the invalidation exists as bookkeeping and not as
behaviour, which is the same as not existing for your agent mid-task.
Two. Nothing fires it.
mark_reviewis a parameter someone passes afterthey already noticed the dependency changed. That is not invalidation by event.
That is invalidation by somebody remembering, which is the thing the memory was
supposed to replace.
Three, and this is the one that made me wince. The dependency index is
written with a 90-day expiry. The mechanism for invalidating by event is itself
invalidated by age. Another commenter in this thread had just pushed back on
that same TTL on the history side; I did not know it also ate this.
So: contradiction is resolved at read time, by whoever is reading, exactly as
you assumed - and the write-time machinery that would have prevented that is
built up to the last inch and then stops.
On "a reason to die instead of a confidence score" - I am taking that whole,
and not only for the memory.
Your sentence about the agent mid-task ("it picks one and keeps going, and you
find out later") describes a failure I hit three times today, in three
systems that have nothing to do with each other. Same shape every time: no
state for "I cannot say", so something plausible gets substituted.
The ranker. Features are normalised across the candidate pool. A missing
value became 0 - the worst value, not a neutral one. 108 records were
being penalised for a field they were never supposed to have. I measured
with the feature and without, concluded it was harmful, and removed it. The
measurement was right. The conclusion was wrong.
A health endpoint. An instance with zero records was classified "100 %,
healthy" - the best value. Paired with a random sample of 8 out of 67
instances, mostly empty test accounts, that made ten consecutive calls eight
seconds apart come back five times "fine" and five times "outage". Our alerts
had been flapping for two nights and I had already fixed a different, real
cause one layer up. This one was underneath it.
My own tooling. I wrote a contrast checker for our UI that hardcoded a
white background. Run against a dark app it produced 341 findings, every one
of them with a line number and two decimal places, every one of them wrong.
Had I not checked, I would have "fixed" a working interface.
Three systems, three substituted defaults, three different directions - worst,
best, and assumed. The common part is not the value. It is that none of them
could say "no statement". Your framing names the fix better than mine did:
a fact needs a reason to die, and a system needs a way to say it has nothing
to report. Those turn out to be the same missing thing seen from two ends.
The health endpoint now has three states instead of two —
aus(recordspresent, index gone),
leer(answered, nothing to say),nicht_gemessen(didnot answer). Keeping
leerandnicht_gemessenapart matters more than itlooks: collapsing them turns the common case into an alarm, which is worse than
the flapping was.
What I owe you, in order, and I would rather write it down than let it stay a
preference:
needs_reviewhas to be read at rank time, or the flag is theatre.The first is small and I had not seen it until you asked. Thanks for pointing
at the write path - I would have kept polishing the read side.
The three-states fix is the one I'd defend hardest, and payments will back you up on it. "No payment exists" and "we could not reach the bank" look identical from the caller's side and mean opposite things. Collapse them and the retry logic does the wrong thing at the worst possible moment, because a retry against nothing-happened is free and a retry against we-don't-know is how someone gets charged twice.
On
needs_reviewat rank time, I'd go further than ranking. Ranking still lets the record through, just lower, and an agent mid-task will take the third-best answer without ever noticing it was third-best. What worked for us was making the stale record refuse to serve rather than serve quietly with a worse score. Loud and useless beats quiet and plausible. You can always add an override for the caller who genuinely wants the last known value and says so out loud.The TTL on the dependency index is the one I'd fix first though, ahead of both. Not because it's the biggest, but because it's the one that fails silently on a schedule you didn't choose. Day 89 everything works. Day 91 the invalidation stops firing and nothing anywhere changes shape. Every bug I've had that waits three months to show up got shipped by someone confident, and most of those times it was me.
Payments as the argument for three states is the strongest version of it - "retry against nothing-happened is free, retry against we-don't-know is how someone gets charged twice" compresses the whole design into one sentence. Taken. On refuse-to-serve versus serve-with-a-warning, we landed one notch away from you, deliberately, and the reason is specific to memory: a superseded record's history is itself information - why it was replaced is often the answer's most useful half. So our stale record serves loudly: the banner is the first line of the text the agent actually reads, not metadata it can skip, and it points at the successor by name. Your scenario - the agent silently taking third-best - is real, but it's a rendering failure, not a ranking failure; a warning the caller can't not-read does the refusing at the right layer. Where I'd adopt your version outright is anything transactional: memory can afford a marked ghost, a payment path can't.
Your TTL point I'll take further, because day-91 bugs are a class we keep meeting: anything that changes behavior on calendar time needs a canary that crosses the boundary early and often. For a 90-day TTL: one synthetic entry with a 7-day TTL whose expiry must be observed every week, through the same invalidation path. If the weekly ghost stops dying, the mechanism broke - 84 days before it matters. And one question back on your override ("the caller who genuinely wants the last known value and says so out loud"): do you audit those? Every override design I've shipped drifted toward being the default path within a quarter, precisely because it always works. An override that isn't counted is a refusal that isn't one.
The “prevention, not usage” metric is a really good distinction. Usage tells you the tool is being invoked; preventing repeated mistakes tells you it actually earned its place in the workflow. I’d be curious how that metric changes once you have multiple users with very different memory patterns.
Thanks - and I have to give you the honest answer rather than the good one: I can't tell you yet, because I don't have that data.
The tool has effectively one heavy user, so every number I have about prevention describes one person's memory patterns. I could dress that up as an early finding, but it'd be a description of me, not of the metric.
What I think would actually show up with several different users is the failure mode, not the success: a lesson written in one person's vocabulary that never gets recalled by anyone else, so it counts as "stored" and prevents nothing. Prevention is measured at recall time, and recall depends on the words the asker uses. That's the number I'd want to watch first - the share of stored lessons that never get retrieved by anyone but their author. If that's high, the metric is quietly measuring authorship rather than usefulness.
If you end up running something similar across a team, I'd genuinely like to hear whether that's what breaks first.
“The lifetime itself becomes state” is the part I hadn’t considered. I like the direction of deriving it from the referents rather than making it another constant to maintain.
And yeah, the fact that all three bugs came from questions outside the codebase is probably the most interesting result here. It’s a good reminder that code can validate implementation consistency without validating whether the system still makes sense.
"The lifetime itself becomes state" is the part I keep turning over too, and I think the resolution is smaller than it looked.
A derived lifetime is not extra state. It is the removal of state: today there are two numbers (180 days on the resolution log, 90 on the history it points at), chosen in two files, that were never compared. Derived, there is one number and one rule. The thing that can drift is gone, rather than better tuned.
Where it does become state is the case I raised with @pm25coder: the referent set is not fixed at write time. A pointer can acquire new referents later, so "as long as the longest-lived thing it points to" is a moving target. My current answer is that this is not a wrinkle but the same pattern showing up twice - re-derive on every write that touches the entry, which is exactly what our silent-failure record already does with "renewed on write". So the honest form of the rule is renewed on write, not computed at creation.
On your last point, which is the one I find hardest to argue with: three bugs, all found by questions from outside, none needing anyone to read the code. I've now hit the same shape twice more in two days, and both times from the same direction.
One of them is worth spelling out because it is the cleanest example I have. A harvesting tool of mine ran across sixteen public repositories. The first five returned 260–277 pairs each. The remaining ten returned zero, reporting "no issue has a linked PR" - for repositories with 60,000 linked pairs between them. The cause was an hour-rate limit, one line long:
Every error became the statement "this issue has no linked PR". The tool then wrote a valid, empty file, printed "done" and exited 0. No test could have caught it, because nothing was inconsistent: the code did exactly what it said, the file was well-formed, the exit code was correct.
It is the same family as the TTL finds. There was no state for "not measured", so silence was booked as zero - and a zero looks like a result. That is the sentence I'd offer as the generalisation of your point: a codebase can check that it is consistent with itself; it cannot check that its silences mean what it thinks they mean. Someone from outside asking "why is that number zero?" is the only instrument I've found for it.
The fix has three states now - pair, no-PR, not-measured - and it refuses to write a file at all when the third one is non-zero. A half harvest is worse than none, because it looks like a whole one.
Counting prevented rework is the right direction, but a non-empty recall is not yet a prevention event. It can be irrelevant, stale, or confidently wrong—and those false positives are more expensive than a miss.
I’d log a second signal after the task: was the memory cited in an action, did the user accept/correct it, and did it reduce retries or time-to-resolution versus comparable misses? Even a lightweight “used / ignored / contradicted” label gives much better evidence than HIT/MISS alone.
The product boundary also introduces memory authority: provenance, observed-at and valid-from dates, supersession links, confidence, and scope (user/team/repository/environment). Retrieval should prefer current applicable facts, while contradictions should be surfaced rather than blended.
The strongest metric may be net prevented cost: confirmed useful recalls minus corrections and incidents caused by bad recalls. That keeps optimization focused on trustworthy memory, not simply more memory.
You are right about the snippet, and it is the weakest thing in the piece. HIT/MISS counts delivery and I labelled it prevention. Those are not the same, and a non-empty recall that is stale or confidently wrong is worse than an empty one, because it costs the hour it takes to find out.
The part I want to ask you about, because I think it is where this gets hard: "used / ignored / contradicted" — who applies the label?
If the tool labels its own recall as "used", that is a self-reported exit code. It is cheap to collect and it will be systematically generous, because the thing being asked whether it helped is the thing whose helpfulness is in question.
So I have been trying to split your three into what a machine can observe without taking anyone's word for it:
Three honest fields beat four with one that flatters. But if you have seen a way to earn the fourth without self-report, that is the thing I would most like to be wrong about.
Your provenance list reads as a gap list from here. We have author, timestamp, confidence and an audit trail. We do not have valid-from or supersession links — so "prefer the current applicable fact" is not something our retrieval can express. It blends, exactly as you say it should not. That one I can act on directly.
Net prevented cost is the sentence I will end up quoting. Our counting has no term for harm at all. A recall that sends someone the wrong way for an hour scores the same as one that was never made — which means the metric rewards more memory rather than trustworthy memory, and I did not see that until you wrote it down.
Second time you have handed me something sharper than the post it is under - thank you.
I knew what an empty result meant. A new user reads an empty result as a broken tool. Man, that exact realization gets every founder who turns an internal tool into a SaaS. When you're the only user, you subconsciously tolerate horrible UX and silent errors because you know what's happening under the hood. Forcing yourself to hand it to a stranger without explaining a thing is the ultimate reality check.
"A new user reads an empty result as a broken tool" - I've now hit the machine version of the same sentence, and it's worse than the UX one.
Yesterday a harvesting tool of mine ran across sixteen public repositories. The first five returned 260–277 results each. The remaining ten returned zero, reporting "no issue has a linked pull request" - for repositories with 60,000 linked pairs between them.
The cause was an hour rate limit. The bug was one line:
Every error became the statement "this one has nothing". The tool then wrote a valid, empty file, printed "done" and exited 0.
Here's your point, one layer down: I knew what an empty result meant, so I never gave the code a way to say it didn't know. There was a state for "found nothing" and a state for "found something" - and no state for "did not measure". Silence had to become one of the two, so it became zero. And a zero looks like a result.
The founder tolerance you describe isn't only about UX. It's in the data model. When you're the only reader, "no rows" is a sentence you finish in your head, so the schema never learns to finish it. The fix wasn't better error handling - it was adding the third state, and refusing to write a file at all when it's non-zero. A half harvest is worse than none, because it looks like a whole one.
Your last line is the one I'd put on the wall: handing it to a stranger without explaining a thing. That's also the only way I found this — a colleague asked "why is that number zero?" and I had no answer that wasn't "huh".
correcting your own headline number unprompted is worth more than the number was.
on measurement though. self-reported precision@1 against your own corpus is always going to be soft, you wrote the queries and the ground truth. locomo and longmemeval exist for exactly this and they're mean in a useful way. multi session, contradictory facts, temporal reasoning. gives you a number somebody else can reproduce.
with a 49 percent overwrite rate i'd bet either one surfaces your contradiction handling gap in week one instead of month three.
HIT/MISS instinct is right but reidmarlow is more right. prevention is "did the action change". that's a diff, not a retrieval metric.
i co-founded backboard, we're top of both those benchmarks so weight it however you want. our locomo eval setup is public if it's useful as a starting point: github.com/Backboard-io/Backboard-...
You're right, and I'd rather say so directly than negotiate.
"You wrote the queries and the ground truth" is the correct objection to every number in that post. 499 lessons I wrote, 100 questions I wrote, one store, one person. That's not a benchmark, it's a self-portrait.
Here's what I did about half of it, finished today. A closed GitHub issue with a linked, merged pull request is exactly a (query, answer) pair - and the person who wrote the complaint is never the person who wrote the fix, and neither is me. I harvested 16,266 of them from 43 public repositories across Go, Rust, Python, TypeScript, Java, C++ and PHP: gitea, grafana, terraform, cockroach, deno, rust-analyzer, tokio, airflow, webpack, elasticsearch, pandas, electron and thirty more.
The obvious trap is a pair whose answer is quoted in its question ("fix timeout in deploy" ← "deploy times out"). Those make any ranker look excellent and prove nothing, so the selection rejects anything above 60% word overlap outright. Measured on the result: mean overlap to its own gold answer 16.6%, to a random other answer 4.4% - the distance is the statement. One pair out of 16,266 above 75%.
That fixes the authorship of the queries. It does not fix your actual point, which is reproducibility - a corpus I harvested with a tool I wrote is still my instrument. So:
On LoCoMo and LongMemEval. I haven't run either, and I'd been treating that as reasonable because they measure conversational memory over multi-session dialogue and my task is retrieval over an unordered store of technical lessons. Having actually looked rather than assumed: that reasoning holds for maybe half of it and is an excuse for the rest.
LongMemEval's knowledge-updates and abstention categories are not conversation-specific. They're the two things my store demonstrably cannot do:
Knowledge updates. A lesson is keyed by topic, so a correction under the same topic replaces it - but a correction under a different topic name leaves both alive, competing on relevance, with no supersession link and no valid-from date. Your 49%-overwrite prediction is well aimed: 257 of 524 lessons have been overwritten at least once, 408 overwrites, 638 superseded versions on disk. Every one of those is a chance for the wrong version to win.
Abstention. My store has no way to return "I cannot know this from what I have". Two days ago I found our production watchdog doing exactly that in a different context: emitting a verdict where the only honest answer was "I can't tell from this sample". It had no state for it, so it said something else. A retrieval layer with no abstention state has the same defect and I hadn't connected the two.
So the plan, in order, and I'd rather commit to it in public: LongMemEval first, because those two categories test the specific gaps I already know about and it produces a number someone else can reproduce. LoCoMo after, as the harder multi-hop and temporal case. My 16,266-pair corpus stays as the domain-specific counterpart - it answers "does this generalise across 43 vocabularies", which neither of those asks.
On prevention being a diff, not a retrieval metric - yes, and Reid was right ahead of me. The only design I've found that isn't a self-report is holding back the top record on a random half of eligible turns and comparing outcomes in aggregate. Expensive, in that half my own sessions get a deliberately worse answer. I haven't found a cheaper one that isn't the tool grading its own homework.
Thanks for the disclosure, and for the link. A competitor pointing at the benchmark that will make me look worst is a more useful comment than the ones agreeing with me, and I'd have taken longer to get there alone.
“Count prevention, not usage” is such a strong way to frame an internal tool. I also liked the honesty around empty results: the builder understands silence, while a new user reads it as failure. That is one of those small product lessons that applies far beyond MCP. Really thoughtful write-up.
Thank you - "count prevention, not usage" took me six weeks to arrive at, mostly by watching the usage numbers stay flat while the tool was quietly saving me time. The empty-results point you picked out is the one I keep chewing on: the builder knows an empty answer means "nothing relevant stored", but a new user reads it as "the tool failed." I haven't fully solved it. Current best attempt is making the empty state say what it looked through ("searched 499 lessons, none about X - want to save this one?"), so silence becomes evidence instead of absence. If you've seen a product that handles empty results really well, I'd genuinely like to know which one - it seems to be an underdesigned corner everywhere.
Interesting experiment! Building an MCP memory server around real usage for six weeks should provide some valuable insights into what information is actually worth persisting versus what becomes noise. I’d be especially interested in how you handled memory retrieval, context relevance, and preventing outdated memories from influencing responses.
Thank you - and you picked the three parts I have the most and the least to say about, in that order.
What is worth persisting versus what becomes noise. This is the one where six weeks of being the only user actually produced a number. Our own automatic capture - turning git commits into records at the end of a session - was generating about 21% noise. The cause was not the idea, it was which half of the commit we read. We took the subject line. The subject says what changed; the body says why it was wrong before. So we were persisting the least informative half of the best-labelled data in the repo, and doing it reliably.
A second one from the same corner: noise is recognisable by fields, not by name. When I cleaned up machine-generated entries, the tempting move was to delete by topic prefix. That would have taken real records with it - the prefix was shared. The reliable signals were structural: which writer produced it, whether it was auto-captured, which tags it carried. Anything that looks like a naming convention will eventually be used by a human for something else.
Retrieval and relevance. Word matching alone gets us roughly a quarter of questions into the top three; adding a semantic pass roughly doubles that. But the honest part of this answer is a correction I posted further up this thread: I published a retrieval number that turned out to be a hardcoded string in our own output, with a green test guarding the sentence rather than the measurement. So I would rather point you at that comment than hand you a fresh figure here. The rule I took from it: a number the server does not compute cannot fall, which means it is not measuring anything.
One finding from the same rebuild that is more useful than any percentage. Our session briefing shows roughly the first 100 characters of each record. We measured where the decisive fact actually sits in the text: typically somewhere between character 300 and 1500. So the briefing was reliably showing titles and calling it knowledge. The fix was not code - it was a writing rule that the one thing you would want at a glance goes in the first sentence. Retrieval that finds the right record and then shows the wrong part of it fails in a way no retrieval metric catches.
Preventing outdated memories from influencing responses. This is the weakest of the three and it has been taken apart properly further up this thread by two other commenters, so I will point rather than repeat: we can write down that a record depends on something that may change, and we do - but that flag is never read at ranking time, so a record marked as needing review is returned exactly as confidently as one that is not. There is also no valid-from date, so retrieval cannot prefer currently applicable over still plausible.
Those threads are the better answer to your third question than anything I would write fresh, mostly because the framing in them is not mine.
Great job
Thank you
Thanks