Building in public
You know the risk before you start. Everyone tells you: do not build something the platform could ship. You build it anyway, because you need it and nobody has it. Then one Tuesday the release notes arrive.
What the months actually looked like
I want to be precise about the cost, because the cost is the reason the release notes hit the way they did.
Two hours of sleep on a normal night — not one heroic week, the normal shape of the last few months. Work during the day, build in the evening, debug until the birds started. Weekends were the good days, because nobody interrupted.
What got built in that time: a memory layer for AI coding assistants. It saves what was learned after a fix and reads the relevant parts back before the next task. It runs over MCP, so it works in whatever editor you happen to open. It survives restarts, model upgrades and switching tools.
I did not build it as a business idea. I built it because I was tired of explaining my own four servers to an assistant every single morning.
The hour the release notes landed
Anthropic shipped memory into Claude Code. Not "context improvements", not "a longer window". The word in the release notes was memory — the same word I had been using for months to describe the thing I was building.
I read it twice. Then I sat there and did the arithmetic that everybody in that position does: months of evenings, the sleep, the weekends — against one line in someone else's changelog.
The thought was not complicated. It was three words long. Who needs mine?
If you have never had a platform ship your feature, the closest thing I can describe is finding out the thing you have been carrying uphill was already at the top. Not that it was wrong. That it was unnecessary.
I did not open the editor that evening. That is the honest version. I read the docs, I read them again, and I went to bed early for the first time in months, which is a strange way for a bad day to end.
Why the fear was rational, not dramatic
It would be easy to write this as an overreaction, and it was not one. The fear had a clear structure, and naming it is useful for anyone standing where I stood.
Distribution. A feature inside the tool wins by default. It is there when you install, it needs no account, it needs no configuration. Mine needed a decision from the user.
Trust. A memory that holds your codebase knowledge is not a small thing to hand to a stranger. The vendor already has your code in the context window. I had to earn what they already had.
They know their own harness. Anything I do from the outside is a guest in someone else's house, and the house rules can change in a release.
None of those three went away, by the way. They are still true. This is not a story where the threat turns out to be imaginary.
The question I could not answer that night
Here is where the evening actually turned, and it was not optimism. It was a question I could not answer, which is a much better reason to keep going than hope is.
I had spent months building this thing, and I could not say — out loud, in one sentence — what it did that the vendor's version did not.
Not because there was no answer. Because I had never needed one. My use case was my four servers and my own forgetfulness. I had never once asked what the thing was for beyond me.
That is a genuinely uncomfortable position: you cannot tell whether you have been made redundant, because you never described what you were in the first place.
So I did the only thing that produces an answer instead of a feeling. I wrote a test.
The test you can run on any memory, including your own
It asks one question: does a fact learned in one session come back in the next one, without help?
Teach it something only true in your world — a fact it must have stored, not one it could guess. Then close everything, come back, and ask.
Write down what you expected before you run it. That part matters, because a memory that returns something plausible feels like a hit until you check it against what you actually stored.
And ask several times across several fresh sessions. One success is an anecdote. A ratio is a measurement.
#!/usr/bin/env bash
# Does a fact survive between sessions? Ratio, not anecdote.
# ./memtest.sh 10
set -uo pipefail
RUNS="${1:-10}"
QUESTION="what is the deploy host for the staging box?"
EXPECTED="10.8.0.7" # the value YOU stored — write it down BEFORE running
hits=0
for i in $(seq 1 "$RUNS"); do
# A fresh process per run. A warm session proves nothing: the fact may
# still be sitting in the context window rather than in any memory.
out=$(claude -p "$QUESTION" 2>/dev/null)
if grep -qiF -- "$EXPECTED" <<<"$out"; then
hits=$((hits + 1))
else
printf 'miss %2d: %s\n' "$i" "$(head -c 100 <<<"$out" | tr '\n' ' ')"
fi
done
printf '\nrecall: %d/%d\n' "$hits" "$RUNS"
# 10/10 means it stored the fact. 6/10 means it stored something else.
# 0/10 with a confident-sounding answer is the worst result and the most
# common one — that is a guess wearing a memory's clothes.
The -p flag is the whole trick: it starts a new process every time. A warm session proves nothing, because the fact may still be sitting in the context window rather than in any memory at all.
Swap claude -p for whatever your assistant's one-shot mode is. Run it against the vendor's memory. Run it against anything you built. The number is the same kind of number, which is the entire point.
What changes for you
Before: you hear that a platform shipped your feature and you decide how you feel about it, using the same information the announcement gave you. That is a decision made on a press release.
After: you have a ratio for both, produced by the same script, and the decision to keep going or stop is made on the difference between two numbers.
Because when I finally ran the numbers, the result was not the one I had spent the evening dreading. It was stranger than that.
Here is where I have to tell the embarrassing part, because the first version of this post had a different table in it.
I had a benchmark. Seventeen lessons, thirteen queries, hand-written. It said our ranker scored 92.3% precision at rank 1 against 76.9% for a flat file. I nearly published that.
Then I ran the same two versions against 498 real lessons from my own brain, and the order flipped. The version the small benchmark preferred by 23 points found the right answer half as often on real data — 15% against 30%. The entire advantage on the fixture corpus was the damage on real data.
Seventeen lessons is not a noisy version of five hundred. It is a different question. With sixteen competitors, a rare word is enough to win, so every ranking mechanism clears the bar and they all look equally good. The differences are invisible, not small.
So here are the numbers from the corpus that is actually mine: 499 lessons, 100 questions, written in plain language, run against three rankers over the same data.
words only + error-text doors cachly vs words
Right answer #1 21.0% 38.0% 40.0% +19.0
In the top 3 27.0% 52.0% 55.0% +28.0
In the top 10 44.0% 69.0% 72.0% +28.0
In the pool at all 78.0% 97.0% 97.0% +19.0
"Words only" is term overlap with no semantics — roughly what an assistant does when it greps a memory file. It puts the right lesson first 21% of the time. Ours does it 40% of the time.
Read that second number without any enthusiasm, because it deserves none: six times out of ten, the first thing we hand back is not the right one. That is the honest state of the art in my corpus, and anyone quoting you a retrieval number above 90% on real data is measuring something other than what you think.
The gap is the point, not the level. Nineteen points on rank 1, twenty-eight on the top three. And the last row is where the fear from that evening finally had an answer: 97% of the time the right lesson is somewhere in the candidate pool. A flat file gets it into the pool 78% of the time and then hands the assistant everything at once.
That row is what the difference actually is, and it is not "mine is better". A file-based memory hands over everything and lets the model sort. Mine sorts first and hands over three things. Which half you need depends on something nobody advertises: how wide the window is at the moment the assistant reads. Ours is 100 characters in the session briefing. At that width, "it is in there somewhere" and "it is first" are not the same result at all — and I had been the last person to notice, about my own product.
The full story of how a benchmark told me the wrong thing for weeks, and the three mistakes I made after I found out, is its own post.
What the first post did, and what seven strangers changed
I published the first version of this story a few days ago: I built an MCP memory server for one user (me, for six weeks).
Twenty-seven comments. The eleven posts before it, over the ten days since my first one, drew eighteen between them — so this single thread outweighed everything else put together. That is not "went viral". It is the difference between publishing into silence and having twenty-seven people argue with your instrument.
I want to be specific about what came out of it, because "great feedback, thanks all" is the sound of nothing happening.
Mads Hansen (@mads_hansen_27b33ebfee4c9) put a hole in my main metric in two sentences: a non-empty recall is not yet a prevention event — it can be irrelevant, stale, or confidently wrong. That is now a rule I apply well beyond the memory. Two days later I found our own production watchdog doing exactly that: returning a verdict where the only honest answer was "I cannot know from this". It had no way to say so, so it said something else.
Reid Marlow (@reidmarlow) asked for a harsher metric than recall count. Abhiix0 (@abhiix0) and Suraj Suradkar (@suraj09) both pushed on prevention versus usage — and Suraj asked the one I could not answer: what happens to that metric with several users who have different memory patterns? The honest reply was that I do not have several users. That limitation is now the fourth paragraph of everything I write, instead of a footnote.
pm25coder (@pm25coder) did the thing I did not expect. He proposed a rule about when a timer is legitimate in a data structure. I turned it into a test rather than agreeing with it, ran it across the codebase, and it found a case the rule did not cover — plus a second-order bug underneath: a pointer that outlived the things it pointed at, because two lifetimes had been chosen in two different files and never compared. His refinement was better than my finding: a timer is legitimate when it defines what the structure means, and wrong when it merely takes it away. And then: make the lifetime derived, not set, so there is no second constant to drift.
arun rajkumar (@mickyarun) named the one-user phase as a design phase rather than an embarrassment — you get to change the schema on a Tuesday, because you are the only person who would notice.
Saleha Mubeen (@saleha_mubeen_aeed05ee62b) asked what information actually turns out to be worth keeping after six weeks. I did not have a number for that either, and it is the question underneath the whole measurement section above.
All of the following was built in the two days after that thread. None of it was on my list before it:
- The benchmark now counts how many questions moved, not how many percentage points changed. On 100 questions, "four points better" is four questions. Twenty-five better against five worse is a result. Fourteen better against thirty worse is the same four points and a disaster.
- That counter immediately killed my most promising change. On the corpus I had been developing against, it moved the top-3 rate from 58% to 63%. On the store that actually runs in production, the same code moved it from 55% to 51%. Same change, same metric, opposite sign — and the second corpus was the real one.
- The watchdog that judged a fleet of 68 from a random sample of 8, and reported it as a fact about all 68. It now measures all of them, and the response carries a flag saying whether the verdict saw everything.
- Three silent
catchblocks that swallowed the reason embeddings failed to write. They still do not fail loudly — a lesson must save even without a network — but now they say why, once per process.
That last one is the whole thread in miniature. The failure was never silent. Only its cause was, and nobody had noticed for two days.
I build cachly — memory for AI coding assistants, over MCP.
ChatGPT and Claude remember your conversations. cachly remembers your system: the bug you fixed, why you chose Postgres, the deploy step that always breaks — and which decision it contradicts. Every assistant you use reads the same memory, and every lesson carries the name of whoever learned it, so nobody from your team has to learn it twice.
Free tier, hosted in the EU: cachly.dev
Top comments (2)
We hit a similar issue on a supplier risk project using Claude Sonnet as the backend. After a minor version bump, our memory recall format silently returned empty strings for the "reasoning" field and we lost two days tracing a retrieval regression to a system prompt change on their end. What actually fixed it was that we'd been logging the author and timestamp for every memory write. I'm not sure cachly handles attribution the same way when writes come from an automated pipeline, but that's the specific question I'd check before going multi-tenant.
That's a rough two days, and thank you for writing it down - a silently empty field after a version bump is the kind of thing people usually just fix and never mention.
To your actual question: every write carries a timestamp and appends to an immutable audit trail, so an overwritten entry keeps the shape of what it used to say. Author resolution is a fallback chain — explicitly passed, then an environment variable, then git user.name, then the local part of git user.email, then empty with an unknown marker. There's an explicit opt-out too, so a write can be deliberately unattributed rather than accidentally so. And the assistant doesn't add itself as co-author unless you turn that on, because the number worth having is "how many lessons have a human author".
Your instinct about the pipeline case is the right one, though, and I'd rather say so than sell you the happy version: on a CI runner git user.name resolves - to the runner. The chain has an "I don't know" state and never reaches it there, because a plausible wrong answer arrives first. Same shape as your empty reasoning field, only worse in one way: an empty field eventually announces itself, author: ci-runner-04 never will. That's on my list now because of your comment.
Slightly uncomfortable timing on my end: earlier today I wrote an entry with a malformed closing tag, which pushed one field's content into another and left the second empty. The response echoed the broken text back and looked like a success. I did it twice before noticing. The fix repairs it at the write boundary rather than filtering on read - filtering on read saves the retrieval and leaves the stored record wrong, which sounds like the state you spent two days inside.
If you're heading multi-tenant, the thing I'd check alongside attribution is whether your write path can reject a malformed record rather than storing it and reporting success. Mine couldn't until this afternoon.