DEV Community: Heinrich Neb

I tested Claude Code's memory against mine. They are not doing the same job.

Heinrich Neb — Tue, 01 Sep 2026 20:40:47 +0000

Proof over claim

Last time I described the hour a platform shipped the feature I had spent months building. This is the part where I ran the test instead of the emotions, and found two things that were never competing.

One number up front, because it qualifies everything below: this is still one user. Every figure here comes from my corpus, my questions, my four servers. The method transfers. Whether the result does, I cannot tell you yet.

The same test, two different answers

The test from last time is one question: teach it a fact only true in your world, close everything, come back in a fresh session, and count how often the fact comes back.

Both passed. That was the first surprise, and it is worth saying clearly: the built-in memory works. This is not a post about a competitor being bad.

The difference showed up when I changed one thing in the test. I asked the fact from a different editor.

And then again from a different machine. And then I asked a colleague to ask it.

That is where the two answers stopped matching, and it had nothing to do with quality. It had to do with what each thing considers its own boundary.

Session memory and project memory are different products

Here is the distinction I could not put into words on that bad evening, and it took a test to produce it.

A vendor's memory is bound to the vendor's harness. It makes one assistant continuous with itself. That is genuinely valuable and it is what most people mean when they ask for memory.

What I had built is bound to the repository instead. It makes the knowledge about a codebase continuous — across editors, across machines, across people, across model upgrades.

Those are not two implementations of one feature. They are answers to two different questions. "What did I just say?" and "what does this project know?" only look similar until you switch tools.

The clearest way to see it: when someone leaves your team, a session memory leaves with them. A project memory does not, because it was never theirs.

The boundary a similarity search cannot cross

The four boundaries above are about where memory lives. There is a fifth one, and it is about how you find anything in it — I only understood it because someone else wrote it down.

Think about the two entries that matter most together: an error, and the fix that was found three weeks later. Write them out.

deploy hangs at "Build image", worker log says nothing

the runner disk was full; docker prune and restart the service

Those two share almost no vocabulary. One is a symptom, one is a cause; one is about a build step, one is about a disk. In embedding space they sit far apart — not slightly, structurally. And they are the single most valuable pair in the whole store, because together they are the answer and apart they are two anecdotes.

A similarity search cannot connect them. Not because the embeddings are bad. Because semantic closeness and causal connection are different relations, and only one of them is what a vector index measures.

That is why our store keeps a causal path beside the similarity one — which entry led to which, which contradicts which. I had treated that as a secondary feature for a long time. It is the part that similarity cannot do, and I needed an outside article to see it.

This also cuts against my own product, so I will say it plainly: if what you need is "find me the thing that sounds like this", a vector search over a file does that, and the extra machinery earns nothing.

The things my own tool did that I had not noticed

This is the embarrassing part, and it is the reason I am writing it down. I had built features I did not use, because my own use case was one person and four servers.

It can hand knowledge between people. I had built the sharing path months earlier for a technical reason and never once used it, because I work alone on this. For a team it is the entire point: one person debugs a thing at 2 a.m., everyone else inherits the reason.

It can answer why, not just what. Every entry carries what worked and what failed. I had been reading only the first field for months. The second one is where the expensive knowledge lives — the approach that looked right and was not.

It can be asked from things that are not editors. Because it speaks MCP and HTTP, our own operations dashboard queries it: a question box that answers from the event stream plus the stored lessons. I built that as a side project and it turned out to be a second product surface.

And it does not care which model you use. The lessons written by one model are read by the next one. I had treated that as an implementation detail. It is the reason the store survives an upgrade cycle that rewrites everything else.

Where the vendor's version is simply better

A comparison that only finds advantages is an advertisement, so here is the other direction, and I mean it.

Setup: theirs is zero. It exists the moment you install. Mine needed a decision, an account and configuration — and measuring that gap honestly is what pushed us to a trial that needs no sign-up at all.

Depth inside one conversation: theirs sits in the harness and sees everything. Anything from the outside sees what it is told, and that is a real ceiling, not a temporary one.

Trust: the vendor already holds your code. Handing the same knowledge to a second party is an extra decision, and "it is hosted in the EU" is an answer, not a dismissal of the question.

If your work lives in one assistant, on one machine, alone, the built-in memory is the right choice and I would tell you so.

Measure it yourself instead of believing either of us

The test that produced all of this is four variations of one question, and the variations are the whole trick. Same fact, different boundary.

Run it in the tool you taught the fact to. Then in a second editor. Then on a second machine. Then have someone else ask.

Each of those four is a boundary a memory either crosses or does not, and no announcement will tell you which — the crossing is the thing that got designed, and it rarely shows up in release notes.

Write the four numbers down before you form an opinion. The shape of the four tells you which product you are holding.

The harness, extended from last time:

#!/usr/bin/env bash
# Four boundaries, one fact. The SHAPE of the four numbers is the answer.
set -u
FACT="which port the staging database listens on"
EXPECTED="5433"           # what you stored, written down BEFORE you ask

ask() {  # ask <label> <command...>
  local label="$1"; shift
  local hits=0
  for i in 1 2 3 4 5; do
    out=$("$@" 2>/dev/null)
    grep -qF -- "$EXPECTED" <<<"$out" && hits=$((hits + 1))
  done
  printf '%-22s %d/5\n' "$label" "$hits"
}

ask "same tool"      your-assistant --new-session --ask "$FACT"
ask "other editor"   other-editor-cli --ask "$FACT"
ask "other machine"  ssh other-box  your-assistant --ask "$FACT"
# The fourth one is not scriptable, and that is the point:
# ask a colleague to run the same question on their own machine.
# ask "other person"  ...

# 5/5 0/5 0/5 0/5  -> session memory. Continuous with itself.
# 5/5 5/5 5/5 5/5  -> project memory. Continuous with the repository.
# Neither is wrong. They are answers to different questions, and you now
# know which one you have.

One warning from our own numbers: do not measure recall quality by whether the answer sounds right. We once had a stored entry containing the exact address we needed, displayed at session start, and made the mistake anyway — the preview cut off at a hundred characters and the address sat at character three hundred and twenty-three. Delivery is a separate measurement from storage, and it is the one that decides whether any of this pays.

What changes for you

Before: a platform ships something adjacent to your work and you decide, from the announcement, whether you are finished. Half the time you are wrong in the pessimistic direction, which costs you the thing you were building.

After: you run four variations of one test, get four numbers, and find out whether you were building the same product or a neighbouring one. It takes an afternoon and it replaces a week of dread.

The thing I actually learned is not about competition. It is that I had never described my own product, and a platform release forced me to — which turned out to be the most useful thing anyone did for it all year.

I build cachly — memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations; cachly remembers your system: the bug you fixed, why you chose Postgres, the deploy step that always breaks. Every assistant reads the same memory, and every lesson carries the name of whoever learned it.

Try it:

30 seconds, no account — npx @cachly-dev/mcp-server@latest demo in any git repo. It reads your log locally and prints what an assistant would already know about the project. After npx fetches the package, the command makes no network calls.
Claude Code plugin — /plugin marketplace add cachly-dev/cachly-mcp, then /plugin install cachly-brain@cachly.
5 minutes, free tier — npx @cachly-dev/mcp-server@latest autopilot writes the MCP configuration for whichever assistant you use.
Or from the web — cachly.dev · free tier, German servers, no credit card.

Your AI Has a Reviewer. Has Anyone Ever Seen It Say No?

Heinrich Neb — Thu, 27 Aug 2026 04:25:16 +0000

Two weeks ago I counted 204 guards in my repos and found that 89 % had never been shown they can fail. I fixed that for a batch of them. Then tonight I filed a data-loss report about 1,000 files that were never lost — and nobody, including me, had a number to check it against.

Last time I wrote here, the number was 204 and 11 %: of the automated checks in my repositories that draw a conclusion, only about one in nine could prove it was capable of failing. @dev_michael's "AI didn't make me a worse coder, it made me a worse reviewer" is what started that count.

So I did the obvious follow-up work: I made a batch of guards prove they can fail. That fix is real and I'll describe it below, because it costs an afternoon and it works.

It also wasn't enough, and I found out the embarrassing way.

Part one: the fix that works

Every reviewer — a hand-written check, an LLM judge, a second agent grading the first — gets one known-bad case wired through the live path. Not a unit test beside the pipeline. The same entry point real work uses.

Our benchmark harness runs three gates per case, and a case that misses any of them does not run at all:

Unsolved state → must go RED. The check must reject the broken repo.
Solved state → must go GREEN. It must accept the fix.
Known-bad → must go RED again. We re-plant the original mistake deliberately; if the check still passes, it never understood the bug.

Gate three is the one that earns its keep. It has caught real breakage — including, twice, in guards written that same week to catch exactly this class of problem.

This week I pushed it further, because a harness is only as good as the cases in it, and inventing cases is where everyone runs out of imagination. @shreyasht killed his own token-optimization project after discovering his best result — 97 % savings — came from the run that did no work at all: the agent asked a clarifying question, stopped, and the dashboard crowned it. A metric with no way to punish doing nothing eventually rewards it. His fix is one sentence: measure per solved task, never per task.

The imagination problem has a cheap answer, and it was sitting in our database: the incident history is the test suite. We took ten real recorded failures — the kind with a "what went wrong" field written by someone annoyed at the time — and had a model punch each one into a runnable check. Ten lessons in, ten runnable cases out, thirty gates, all thirty passed. Zero discarded.

One thing that surprised me: the first attempt failed the internals filter, and it failed because the example we handed the model contained a product name in a comment. The model copied it faithfully. The filter caught it. A guard I had written to be paranoid about human sloppiness caught a machine being obedient instead.

@wrobeltomasz described this same discipline independently in a comment thread while this draft sat unpublished: identify the checks, then run them "in simulation mode to confirm that they can indeed respond to invalid input" — moving verification "from statistics in the README to actual system resilience." Two people who have never met, same gate, same reason. That usually means the pattern is real and not a personal quirk.

Part two: the half I had not fixed

Here is what all of the above proves: on the day I wired it, that reviewer could say no.

It says nothing whatsoever about today.

@mk023 said the correcting sentence in a thread under my last post, and gave me permission to quote it here:

"Don't just test that the guard can fail — test that it is still guarding the thing you think it is guarding."

That completes the model. A green check can be lying in three orthogonal ways: falsifiability — can it go red at all; liveness — can it go red against today's system; and aim — is the boundary it defends still the one that matters. Part one only bought me the first. Marco's sentence is the other two, and the rest of this post is me finding out the hard way that he was right.

@james_anderson_h, in a thread about AI workspaces that ship with "guardrails — approval gates, an audit trail, a second agent that reviews the first," put the gap better than my question had: most tools hand you the audit log, not proof the veto still fires.

"A checker nobody has ever seen fail is indistinguishable from a checker that approves everything. The two produce identical logs right up until the day the rubber-stamp waves through the thing that hurts you."

My answer to him was a small thing I now think is the most useful idea in this post: the veto heartbeat.

Surface one date. "Last time this reviewer refused something: 2 days ago." First-class, visible, right next to the uptime number. The known-bad case runs on a schedule; if the last-refusal date ages past that schedule, staleness itself is the alarm. No log-diving, no trusting a vendor's dashboard, no archaeology. One date anyone can read in one glance — and the absence of fresh red finally looks like what it is.

James named the principle underneath it better than I had:

"Silence and health look identical unless you deliberately build a state for 'not verified lately'."

That is the whole bug in one line. Most systems model pass and fail and nothing else. A veto that quietly died therefore reads exactly like a veto that simply had nothing to refuse. The heartbeat is that missing third state, worn on the outside where an auditor can see it without your permission.

Where I was the punchline, tonight

I was going to publish the section above as the ending. Then I spent an evening proving the point against myself, and the story is better than the theory.

I went to check on a long-running harvest job spread across two machines. I connected to one, looked for the working directory, found nothing, looked for the process, found nothing. So I reported it: the data is gone, roughly a thousand collected items lost, and I filed a ticket saying so.

I had connected to the wrong machine. The numbering I used to pick it does not mean what I assumed it means — a fact that is written down in my own project instructions, in bold, with an example. I had read it before. I did it anyway.

Nothing was lost. The harvest was sitting exactly where it should be, and a second job was quietly running on it at that very moment.

Now the part that matters, because "I made a dumb mistake" is not an article. Why did the false report survive as long as it did? Because there was nothing to contradict it. No dashboard, no counter, no file with three numbers in it. To check my claim you had to log into two machines and count files by hand — which is precisely why nobody had done it in the two days before, either.

And when I finally did count, the numbers exposed something nobody had noticed: the two machines had been working the same list from opposite ends, had met in the middle days ago, and had since re-collected 332 items both already had. Not broken. Not alarming. Just quietly wasteful, in a system with no visible number to be quietly wasteful in front of.

The two failures have exactly one cause

A green check nobody can question and a red alarm nobody can question are the same bug wearing different clothes:

A guard that has silently stopped being able to fail produces unfalsifiable green.
A monitor with no visible state produces unfalsifiable red — my false report, which survived only because contradicting it required work.

Both come from the same missing thing: a number that a stranger can read without your cooperation. The veto heartbeat is that number for a reviewer. Three counters in a file is that number for a background job. Neither is clever. Both are absent almost everywhere, including — until this week — in a codebase belonging to someone who writes about this for a living.

@bert_sk_shim_cb93b1 hit the same class from the opposite side while this was being written, and named the part I had missed. His login check queried an endpoint scoped to a different auth method, so it returned "no username" no matter what — always negative instead of always green. Then the sentence that reframed this whole post for me:

"A red result tends to get remediated rather than investigated, so I had someone go log in again, which was unnecessary, and if the timing had been a bit different I'd have credited that as the fix and kept the broken check."

That is the asymmetry. Green invites complacency, but red invites action — and action feels like resolution. A broken always-red check gets a workaround performed in front of it, and the workaround gets credited. He very nearly filed an unnecessary login as the fix and kept the instrument that lied to him. Always-negative hides better than always-green, and it is the direction almost nobody watches.

His fix generalizes further than either of our stories: separate presence from value. The operational version I have landed on since is one line — print what you read before you print what you concluded. "No username at /whoami (auth scheme B)" is a bug report. "Not logged in" is a rumour with a status code. My false alarm would have died in thirty seconds if my own check had said which machine answered instead of only what I concluded about it.

@pm25coder shipped the same move at a different layer while I was writing this: their harness now appends every silent-safety-net event to an append-only file that survives restarts and log rotation, so "did the safety net ever misfire?" became a lookup instead of archaeology. Evidence that outlives the process that produced it. That is the whole idea, and it is worth stealing.

And John Green has been running the harshest version of this discipline on model evaluations: he reran his exam five times and watched his published winner evaporate — the behaviour that decided the ranking reproduced zero times out of five. His line stays with me: "The exam caught its author again — not in the answer key, not in the grader — in how confidently I read a single run."

Copy that

Paste this next to any pipeline that claims to have guardrails:

REVIEWER CHECKLIST — "exists" is not "works"
[ ] One known-bad case wired through the LIVE path (not a side test)
[ ] It runs on a schedule, not once at setup
[ ] Its last RED has a visible timestamp ("last refusal: …")
[ ] Timestamp older than the schedule => alarm, automatically
[ ] The metric it guards punishes doing nothing (per SOLVED unit)
[ ] Someone has seen every terminal state fire at least once

MONITOR CHECKLIST — an alarm needs a number too
[ ] Three counters, readable without logging into anything
[ ] The identity of the thing measured is IN the output
    (which machine, which provider, which run — not inferred)
[ ] Silence has its own state, distinct from "healthy"
[ ] Every verdict prints the VALUE it read, not just the conclusion
[ ] Check the always-negative direction too: a check that can only
    say no is as broken as one that can only say yes — and it hides
    better, because red gets remediated instead of investigated

The 60-second version, if you do nothing else: open the oldest green check in your pipeline and ask when it last went red. If the answer needs a log search, you have found one. If you cannot answer it at all, you have found a bigger one.

What this does not prove

The three gates prove a check could fail on a case someone thought of. They say nothing about the cases nobody thought of, and harvesting your own incident history — which I recommend — has a bias baked in: those cases come from the same system that produced the failures, so they may be systematically easier than reality. If your harvested cases pass at a noticeably higher rate than your invented ones, that is a finding, not a win. Report the two separately.

The heartbeat has a failure mode too: a scheduled known-bad case can become a ritual that always passes, and then the fresh date is reassuring rather than informative. The honest mitigation is rotation — change the known-bad case periodically — and I am not going to pretend I have a clean answer for how often.

And the counters are not free. Three numbers per job is a small amount of code and a permanent amount of maintenance.

The question I'll leave you with

Find the oldest green checkmark in your pipeline — the linter gate, the LLM judge, the approval step that has passed everything for months.

When did it last say no? Do you know? Can you find out without asking the person who built it?

If the answer is "I would have to check the logs," you do not have a guardrail. You have a rubber stamp with excellent posture.

I would genuinely like to hear two things in the comments: the age of the oldest never-red green in your system, and — the one I now care about more — the last time an alarm in your system turned out to be wrong, and how long it took anyone to prove it.

Mine took an evening, and only because I finally counted.

Quoted with thanks: @mk023, whose one sentence is the whole of part two, @shreyasht's token-stack post-mortem, John Green's AI-verification series, @james_anderson_h's workspace thread, and @wrobeltomasz, @pm25coder and @bert_sk_shim_cb93b1 in the comments — Marco's and James' lines with their explicit blessing. Bert's always-negative direction is the half of this post I would not have found on my own.

I build cachly — memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your system: the bug you fixed, why you chose Postgres, the deploy step that always breaks — and which earlier decision it contradicts. Every assistant you use reads the same memory, and every lesson carries the name of whoever learned it — so nobody has to learn it twice.

The incident-history-as-test-suite trick in this post runs directly on that memory: the lessons were already there, written at the moment someone was annoyed enough to record them.

Free tier, hosted in the EU: cachly.dev

AI promoted every developer to reviewer. Nobody tested the reviewer.

Heinrich Neb — Mon, 24 Aug 2026 23:23:06 +0000

I wanted to disagree with 'AI made me a worse reviewer' from Michael Amachree (@dev_michael) . Instead I counted 204 of my own guards — and 89 % of them have never been asked to prove they can fail.

Michael wrote something that I couldn't put down: AI didn't make me a worse coder, it made me a worse reviewer. Here is the number, and it's worse than his thesis: of the 204 automated checks in my repositories that draw a conclusion, only 22 can prove they are able to fail. That's 11 %. The other 89 % have never once been shown a known-bad input. They are green. Whether they are green because everything is fine, or green because they are incapable of finding anything - I could not have told you last week. And I'm the person who wrote them.

What I actually counted

First the definition, so you can reject it or reuse it.

A conclusion-bearing guard is any test that reads source code, config, or system state and asserts a claim about it. Not "does this function return 4" - but "no workflow downloads its cache over the network", "every page passes the same quarter filter", "this feature flag matches the deployed spec". The tests that stand in for a human reviewer.

A negative control is a probe that feeds that guard a known-bad input and asserts it gets rejected for the expected reason. Our convention marks them KONTROLLE: in the test name.

Counting is mechanical: 204 guard files across three repositories, 22 with at least one control probe, 54 probes total. The counter is a proxy - marker-based, so unmarked controls and false-positive guard files put the true number at plus or minus a few points. The shape survives any correction: most of my reviewers have never been reviewed.

Three green-and-blind checks, one ordinary week

This isn't theoretical. All three of these happened to me in the last seven days, in production tooling.

The deploy gate that died of its own medicine. A pipeline step existed specifically to catch a silent failure mode - a missing tool falling back to an empty result. It called node -e to parse a health response. The deploy runner has no Node. Six consecutive deployments failed with exit 127 - the check against missing tools failed on a missing tool, and nothing shipped for six hours. The step had been green in review because nobody had ever run it where it actually runs.

The harvester that threw away its own work. An autonomous job collected data from public repositories and judged each run by exit code. One run wrote seven perfectly good records, then hit a non-fatal warning and exited non-zero. The machine booked its own completed work as "failed, retry later" - because interrupted-with-partial-results had no representation, only success and failure. We caught it because the result file was sitting on disk right next to the exit code that denied its existence.

The pattern that matched the wrong 500. An error classifier looked for server errors with the pattern 50[024] - anywhere in the output. It matched the "500" inside "4258 of 5000 quota points remaining" and classified a successful run as a server failure. Every field it read was real. It was answering a different question than the one asked.

Three different systems. One shape: the check watched a messenger - an exit code, a pattern, a status - while the artifact that mattered told a different story.

What this has to do with AI making you a worse reviewer

Here's where I think Michael's post lands harder than he says.

AI moved my job. I used to spend most of my day producing artifacts and a little of it verifying them. Now an agent produces most of the artifacts, and my job is verification. Which means my real codebase - the one my judgment actually ships through - is those 204 guards.

And that codebase is held to a standard I would reject in application code. No test coverage (11 %). No review of the reviewer. Green as the default state, silence booked as success.

When Michael says AI made him a worse reviewer, I'd sharpen it: AI promoted us all to reviewers, and none of us tested the reviewer. The model isn't the weak link. The unfalsifiable green checkmark is.

The rule that survived the week

Everything above collapses into one sentence we now apply mechanically:

Judge the artifact, not the messenger.

Exit codes are messengers. Summaries are messengers. The agent's own "done" is a messenger. Green badges are messengers. The artifact is the diff, the file on disk, the served response body, the row in the database. When a messenger and an artifact disagree, the artifact is right - and a check that only ever reads messengers should be treated as unverified, however green it is.

The corollary for guards: a green zero is the most dangerous answer a check can give. "Found no violations" and "is incapable of finding violations" produce identical output. Only a negative control separates them.

Count your own ratio (60 seconds)

This is the part you can use without believing me. Drop this in your repo root - it counts test files that read source or state, and how many carry a marked negative control (adjust the marker to your convention):

// count-controls.mjs — node count-controls.mjs
import { readFileSync, readdirSync, statSync } from "node:fs";
import { join } from "node:path";
const files = [];
(function walk(d) {
  for (const n of readdirSync(d)) {
    if (n === "node_modules" || n === ".git" || n === "dist") continue;
    const p = join(d, n);
    statSync(p).isDirectory() ? walk(p) : /\.test\.(t|j)sx?$/.test(n) && files.push(p);
  }
})(".");
let guards = 0, withControl = 0, probes = 0;
for (const f of files) {
  const t = readFileSync(f, "utf8");
  if (!/readFileSync|readdirSync|execSync/.test(t)) continue; // "reads state" proxy
  guards++;
  const n = (t.match(/KONTROLLE|negative.control|can.?not.?find/gi) ?? []).length;
  if (n) withControl++;
  probes += n;
}
console.log(`${guards} conclusion-bearing guard files · ${withControl} with a negative control (${guards ? Math.round(100 * withControl / guards) : 0} %) · ${probes} probes`);

If your number is above 30 %, I'd genuinely like to know how you got there - that's the discussion I'm hoping for below.

Where I was the punchline, twice, while writing this

Rule 2 of writing these posts is correcting yourself unprompted, so:

While building the feature this article's data comes from, my equivalence test failed by exactly 0.25 - and the bug was in my test, not the code: min-max spreading turns a column of zeros into a column of 0.5s and adds a constant. I had built a probe that answered a different question than the one asked, in the middle of measuring exactly that failure class.

And one push in that same hour went out with a red test - because npm test | grep replaces the test's exit code with grep's. My pipeline read a messenger. The artifact - the failing test - sat right there.

The person telling you to test your reviewers failed to test his reviewer, twice, in one evening. That's not irony. That's the base rate, and it's why conventions beat discipline.

What this does not prove

One developer, three repositories, one week - this is a case series, not a sample. The 11 % is marker-based and approximate. And I have not shown that raising falsifiability coverage improves outcomes downstream; I've shown that at 11 % I couldn't distinguish my working guards from my decorative ones. Whether the number that matters is 30 % or 80 %, I don't know yet - we're raising ours and measuring as we go.

There's also a fair objection: negative controls are themselves tests that can rot. True. But a control that rots fails loudly the next time the guard changes - that's the asymmetry that makes them worth writing.

So: what's your ratio? And more interesting - what's the greenest check in your pipeline that you now suspect has never been able to fail?

Free tier, hosted in the EU: cachly.dev

What does your AI assistant remember from yesterday?

Heinrich Neb — Mon, 24 Aug 2026 10:12:30 +0000

Context that survives

Answer it without opening a chat window. What does your assistant know this morning that it learned yesterday?

For most setups the honest answer is nothing, and nothing has a price. But that is the easy version of the problem. The one that actually cost me an afternoon is worse: it knew, and it did not tell me. The fact was stored, it was displayed, and I made the mistake anyway. More on that below, because it is the part everybody gets wrong on the first try — and the part nobody warns you about.

The cost is a number, not a feeling

Take one week and count the first messages of your sessions. Not all messages, just the opening one of each.

In my own week, seventeen of twenty-two opening messages were setup. Which servers exist, which one runs the database, which deploy path is the real one, which trap I fell into last month. Roughly ten minutes each before any work started.

That is nearly three hours a week spent re-teaching facts that had not changed. Not learning anything. Re-typing.

Before you take that number anywhere: it is one person, one week, twenty-two sessions. It is not a study and I would not defend it as one. The reason I am handing it to you is that it took twenty minutes to produce, and you can produce your own by tomorrow morning. That is the only part of it I would argue for.

The worse half is invisible: the sessions where I did not bother re-explaining, and the assistant confidently used the wrong server name because nobody had told it otherwise.

Why a longer context window does not fix it

A bigger window makes one conversation smarter. It does nothing across conversations, and across conversations is where the work actually lives.

The knowledge you need tomorrow was produced today, at the end of a debugging session, in the moment somebody said out loud why the thing broke. That sentence is the asset. It exists for about thirty seconds and then the window closes.

Documentation is supposed to catch it and mostly does not, because writing docs is a separate task with separate motivation, performed at the exact moment you least want another task.

So the sentence needs to be captured by the participant who is already typing: the assistant.

What changes when the memory outlives the session

Two things get better, and one gets uncomfortable.

Better: the first ten minutes disappear. The assistant reads what it wrote before, and starts from the state of the world instead of from zero. You notice this as the absence of an annoyance, which is a strange kind of win to measure but a real one.

Also better: repeated mistakes get expensive to repeat. When the reason a thing broke is written down where the next session reads it, the second occurrence of that mistake stops being free.

Uncomfortable: you find out how many of your problems were repeats. In my own recorded set, several entries described mistakes I had already made twice. The memory did not make me smarter. It made me stop paying twice.

The part everyone gets wrong on the first try

Storing is easy. Delivering is where this fails, and it fails quietly.

I had a lesson stored that contained the exact address I needed, along with a warning against the exact mistake I was about to make. It was displayed to me at the start of the session. I made the mistake anyway.

The preview showed the first hundred characters. The address sat at character three hundred and twenty-three. The warning lived in a field that was never rendered at all.

The store was perfect. The delivery was a sieve with a hundred-character mesh. If you build this, spend your effort on what gets shown, not on what gets saved.

Build the smallest version this week

You can test the idea in an afternoon without adopting anything. Two habits and a file.

First, at the end of any session where something broke and got fixed, write one line: what broke, what fixed it, which file. One line, not a document.

Second, at the start of the next session, paste the lines that match what you are about to do. That is the whole loop, performed by hand.

Third, and this is the part that decides whether it works: put the runnable thing first in the line. Not the backstory, the command.

A file and two shell functions are enough to try it:

mem=~/.memory.tsv

# save: one line, the fact FIRST, prose after
remember() {
  printf '%s\t%s\n' "$1" "$2" >> "$mem"   # remember "staging-db" "port 5433, NOT 5432 - 5432 is prod"
}

# recall: grep, then read the whole line — not the first 100 chars
recall() {
  grep -i -- "$1" "$mem" | cut -f2
}

# Try it for a week. If recall returns something you would otherwise
# have retyped, you have your answer. If it returns nothing, you learned
# that cheaply.

Two rules make the difference between this working and this rotting. Write the decisive fact in the first ten words. And when you truncate for display, truncate the prose, never the identifiers.

What changes for you

Before: every morning you type the same three paragraphs about your own infrastructure, and on the mornings you skip it, you get confident answers built on the wrong assumptions.

After: the assistant opens with what it learned last time, including the mistake it made and how that got fixed. Your first message is about today's work, not about your server names.

A memory that stores everything and shows a hundred characters is not a memory. It is a filing cabinet nobody can open.

I build cachly — persistent memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your codebase: the bug you fixed, why you chose Postgres, the deploy step that always breaks — including what your teammates learned. And every assistant you use reads the same memory.

Free tier, hosted in the EU: cachly.dev

The best argument against my MCP server came from Anthropic

Heinrich Neb — Sat, 22 Aug 2026 00:27:32 +0000

Building in public

You know the risk before you start. Everyone tells you: do not build something the platform could ship. You build it anyway, because you need it and nobody has it. Then one Tuesday the release notes arrive.

What the months actually looked like

I want to be precise about the cost, because the cost is the reason the release notes hit the way they did.

Two hours of sleep on a normal night â€” not one heroic week, the normal shape of the last few months. Work during the day, build in the evening, debug until the birds started. Weekends were the good days, because nobody interrupted.

What got built in that time: a memory layer for AI coding assistants. It saves what was learned after a fix and reads the relevant parts back before the next task. It runs over MCP, so it works in whatever editor you happen to open. It survives restarts, model upgrades and switching tools.

I did not build it as a business idea. I built it because I was tired of explaining my own four servers to an assistant every single morning.

The hour the release notes landed

Anthropic shipped memory into Claude Code. Not "context improvements", not "a longer window". The word in the release notes was memory â€” the same word I had been using for months to describe the thing I was building.

I read it twice. Then I sat there and did the arithmetic that everybody in that position does: months of evenings, the sleep, the weekends â€” against one line in someone else's changelog.

The thought was not complicated. It was three words long. Who needs mine?

If you have never had a platform ship your feature, the closest thing I can describe is finding out the thing you have been carrying uphill was already at the top. Not that it was wrong. That it was unnecessary.

I did not open the editor that evening. That is the honest version. I read the docs, I read them again, and I went to bed early for the first time in months, which is a strange way for a bad day to end.

Why the fear was rational, not dramatic

It would be easy to write this as an overreaction, and it was not one. The fear had a clear structure, and naming it is useful for anyone standing where I stood.

Distribution. A feature inside the tool wins by default. It is there when you install, it needs no account, it needs no configuration. Mine needed a decision from the user.

Trust. A memory that holds your codebase knowledge is not a small thing to hand to a stranger. The vendor already has your code in the context window. I had to earn what they already had.

They know their own harness. Anything I do from the outside is a guest in someone else's house, and the house rules can change in a release.

None of those three went away, by the way. They are still true. This is not a story where the threat turns out to be imaginary.

The question I could not answer that night

Here is where the evening actually turned, and it was not optimism. It was a question I could not answer, which is a much better reason to keep going than hope is.

I had spent months building this thing, and I could not say â€” out loud, in one sentence â€” what it did that the vendor's version did not.

Not because there was no answer. Because I had never needed one. My use case was my four servers and my own forgetfulness. I had never once asked what the thing was for beyond me.

That is a genuinely uncomfortable position: you cannot tell whether you have been made redundant, because you never described what you were in the first place.

So I did the only thing that produces an answer instead of a feeling. I wrote a test.

The test you can run on any memory, including your own

It asks one question: does a fact learned in one session come back in the next one, without help?

Teach it something only true in your world â€” a fact it must have stored, not one it could guess. Then close everything, come back, and ask.

Write down what you expected before you run it. That part matters, because a memory that returns something plausible feels like a hit until you check it against what you actually stored.

And ask several times across several fresh sessions. One success is an anecdote. A ratio is a measurement.

#!/usr/bin/env bash
# Does a fact survive between sessions? Ratio, not anecdote.
#   ./memtest.sh 10
set -uo pipefail

RUNS="${1:-10}"
QUESTION="which port does the staging database listen on?"
EXPECTED="5433"             # the value YOU stored â€” write it down BEFORE running
hits=0

for i in $(seq 1 "$RUNS"); do
  # A fresh process per run. A warm session proves nothing: the fact may
  # still be sitting in the context window rather than in any memory.
  out=$(claude -p "$QUESTION" 2>/dev/null)

  if grep -qiF -- "$EXPECTED" <<<"$out"; then
    hits=$((hits + 1))
  else
    printf 'miss %2d: %s\n' "$i" "$(head -c 100 <<<"$out" | tr '\n' ' ')"
  fi
done

printf '\nrecall: %d/%d\n' "$hits" "$RUNS"
# 10/10 means it stored the fact. 6/10 means it stored something else.
# 0/10 with a confident-sounding answer is the worst result and the most
# common one â€” that is a guess wearing a memory's clothes.

The -p flag is the whole trick: it starts a new process every time. A warm session proves nothing, because the fact may still be sitting in the context window rather than in any memory at all.

Swap claude -p for whatever your assistant's one-shot mode is. Run it against the vendor's memory. Run it against anything you built. The number is the same kind of number, which is the entire point.

What changes for you

Before: you hear that a platform shipped your feature and you decide how you feel about it, using the same information the announcement gave you. That is a decision made on a press release.

After: you have a ratio for both, produced by the same script, and the decision to keep going or stop is made on the difference between two numbers.

Because when I finally ran the numbers, the result was not the one I had spent the evening dreading. It was stranger than that.

Here is where I have to tell the embarrassing part, because the first version of this post had a different table in it.

I had a benchmark. Seventeen lessons, thirteen queries, hand-written. It said our ranker scored 92.3% precision at rank 1 against 76.9% for a flat file. I nearly published that.

Then I ran the same two versions against 498 real lessons from my own brain, and the order flipped. The version the small benchmark preferred by 23 points found the right answer half as often on real data â€” 15% against 30%. The entire advantage on the fixture corpus was the damage on real data.

Seventeen lessons is not a noisy version of five hundred. It is a different question. With sixteen competitors, a rare word is enough to win, so every ranking mechanism clears the bar and they all look equally good. The differences are invisible, not small.

So here are the numbers from the corpus that is actually mine: 499 lessons, 100 questions, written in plain language, run against three rankers over the same data.

                   words only   + error-text doors   cachly    vs words
  Right answer #1      21.0%          38.0%          40.0%      +19.0
  In the top 3         27.0%          52.0%          55.0%      +28.0
  In the top 10        44.0%          69.0%          72.0%      +28.0
  In the pool at all   78.0%          97.0%          97.0%      +19.0

"Words only" is term overlap with no semantics â€” roughly what an assistant does when it greps a memory file. It puts the right lesson first 21% of the time. Ours does it 40% of the time.

Read that second number without any enthusiasm, because it deserves none: six times out of ten, the first thing we hand back is not the right one. That is the honest state of the art in my corpus, and anyone quoting you a retrieval number above 90% on real data is measuring something other than what you think.

The gap is the point, not the level. Nineteen points on rank 1, twenty-eight on the top three. And the last row is where the fear from that evening finally had an answer: 97% of the time the right lesson is somewhere in the candidate pool. A flat file gets it into the pool 78% of the time and then hands the assistant everything at once.

That row is what the difference actually is, and it is not "mine is better". A file-based memory hands over everything and lets the model sort. Mine sorts first and hands over three things. Which half you need depends on something nobody advertises: how wide the window is at the moment the assistant reads. Ours is 100 characters in the session briefing. At that width, "it is in there somewhere" and "it is first" are not the same result at all â€” and I had been the last person to notice, about my own product.

The full story of how a benchmark told me the wrong thing for weeks, and the three mistakes I made after I found out, is its own post.

What the first post did, and what seven strangers changed

I published the first version of this story a few days ago: I built an MCP memory server for one user (me, for six weeks).

Twenty-seven comments. The eleven posts before it, over the ten days since my first one, drew eighteen between them â€” so this single thread outweighed everything else put together. That is not "went viral". It is the difference between publishing into silence and having twenty-seven people argue with your instrument.

I want to be specific about what came out of it, because "great feedback, thanks all" is the sound of nothing happening.

Mads Hansen (@mads_hansen_27b33ebfee4c9) put a hole in my main metric in two sentences: a non-empty recall is not yet a prevention event â€” it can be irrelevant, stale, or confidently wrong. That is now a rule I apply well beyond the memory. Two days later I found our own production watchdog doing exactly that: returning a verdict where the only honest answer was "I cannot know from this". It had no way to say so, so it said something else.

Reid Marlow (@reidmarlow) asked for a harsher metric than recall count. Abhiix0 (@abhiix0) and Suraj Suradkar (@suraj09) both pushed on prevention versus usage â€” and Suraj asked the one I could not answer: what happens to that metric with several users who have different memory patterns? The honest reply was that I do not have several users. That limitation is now the fourth paragraph of everything I write, instead of a footnote.

pm25coder (@pm25coder) did the thing I did not expect. He proposed a rule about when a timer is legitimate in a data structure. I turned it into a test rather than agreeing with it, ran it across the codebase, and it found a case the rule did not cover â€” plus a second-order bug underneath: a pointer that outlived the things it pointed at, because two lifetimes had been chosen in two different files and never compared. His refinement was better than my finding: a timer is legitimate when it defines what the structure means, and wrong when it merely takes it away. And then: make the lifetime derived, not set, so there is no second constant to drift.

arun rajkumar (@mickyarun) named the one-user phase as a design phase rather than an embarrassment â€” you get to change the schema on a Tuesday, because you are the only person who would notice.

Saleha Mubeen (@saleha_mubeen_aeed05ee62b) asked what information actually turns out to be worth keeping after six weeks. I did not have a number for that either, and it is the question underneath the whole measurement section above.

All of the following was built in the two days after that thread. None of it was on my list before it:

The benchmark now counts how many questions moved, not how many percentage points changed. On 100 questions, "four points better" is four questions. Twenty-five better against five worse is a result. Fourteen better against thirty worse is the same four points and a disaster.
That counter immediately killed my most promising change. On the corpus I had been developing against, it moved the top-3 rate from 58% to 63%. On the store that actually runs in production, the same code moved it from 55% to 51%. Same change, same metric, opposite sign â€” and the second corpus was the real one.
The watchdog that judged a fleet of 68 from a random sample of 8, and reported it as a fact about all 68. It now measures all of them, and the response carries a flag saying whether the verdict saw everything.
Three silent catch blocks that swallowed the reason embeddings failed to write. They still do not fail loudly â€” a lesson must save even without a network â€” but now they say why, once per process.

That last one is the whole thread in miniature. The failure was never silent. Only its cause was, and nobody had noticed for two days.

I build cachly â€” memory for AI coding assistants, over MCP.

ChatGPT and Claude remember your conversations. cachly remembers your system: the bug you fixed, why you chose Postgres, the deploy step that always breaks â€” and which decision it contradicts. Every assistant you use reads the same memory, and every lesson carries the name of whoever learned it, so nobody from your team has to learn it twice.

Free tier, hosted in the EU: cachly.dev

I built an MCP memory server for one user (me, for six weeks)

Heinrich Neb — Thu, 20 Aug 2026 09:05:38 +0000

Building in public

You explain your deploy setup to your assistant. It helps. Tomorrow you explain the same setup again. And the day after. You are not training it. You are re-typing.

The tool nobody asked for

I did not set out to build a product. I set out to stop repeating myself.

My setup is four servers with names that mean nothing to anyone else, a tunnel with a numbering scheme I keep getting wrong, and a dozen small traps that only exist because of decisions I made two years ago. Every new session started from zero.

So I gave the assistant a place to write things down, and a way to read them back before it started working. Two calls: one to save what was learned, one to recall it. That was the whole idea.

For six weeks it had exactly one user. Nobody else could have used it, because I had not written a single line of documentation.

Six weeks of being my own only customer

That stretch turned out to be the most valuable part, and not because of what got built. Because of what got measured.

When you are the only user, every rough edge lands on you within a day. A recall that returns the wrong thing costs you the next hour. A save that silently drops a field costs you the next week, when you go looking for it.

I kept a count of the times the memory actually prevented a mistake. Not a feeling, a count. After six weeks it was high enough that I stopped arguing with myself about whether the thing was worth the effort.

The uncomfortable part: several of those saved lessons were about mistakes I had already made twice. The tool did not make me smarter. It made me stop paying for the same lesson.

The moment it stopped being a personal tool

The thought that changed it was not a market analysis. It was smaller and more honest: if I find this useful, and my setup is not special, then somebody else is retyping their own servers right now.

That is a weak argument on its own. Plenty of internal tools are useful precisely because they fit one person. So I looked for the part that was not about me.

What was not about me: the shape of the problem. Every assistant starts each session with no history. Every developer has context that lives in their head and nowhere a machine can read. That is not my setup. That is the default.

So I wrote the documentation I had skipped, then the onboarding I had never needed, then the parts that only matter when the user is not the author: error messages that explain themselves, a health check, a way to see what the thing actually knows.

What building for one user taught me about building for many

Three things carried over, and one did not.

Carried over: every feature had already survived daily use before a stranger saw it. There was no backlog of ideas nobody had tried. The roughest paths had been walked hundreds of times by someone who could not file a ticket and walk away.

Also carried over: the honesty. When your only user is you, a green checkmark that hides a failure costs you personally, so you stop building those. That habit turned out to be the actual product.

Did not carry over: my tolerance for silence. I knew what an empty result meant. A new user reads an empty result as a broken tool. Half the work of turning it into a product was teaching it to say why nothing came back.

Do this before you decide your internal tool is a product

You do not need my stack for any of this. You need a number and a stranger.

First, count the saves. Instrument your internal tool so it records every time it prevented rework. Not usage, prevention. Usage tells you it runs; prevention tells you it earns.

Second, hand it to one person who did not build it, with no explanation, and watch where they stop. That is your documentation backlog, in priority order, for free.

Third, read your own error messages as if you had never seen the code. Every message that says what happened but not what to do next is a support ticket you have already written.

A quick way to get the count without touching your tool's logic:

# Wrap the recall path and log whether it actually returned something useful.
# Two files, no dependencies, works with any tool that shells out.
log=~/.mytool/prevented.log

recall() {
  out=$(mytool recall "$1")
  if [ -n "$out" ]; then
    printf '%s\tHIT\t%s\n' "$(date -u +%FT%TZ)" "$1" >> "$log"
  else
    printf '%s\tMISS\t%s\n' "$(date -u +%FT%TZ)" "$1" >> "$log"
  fi
  printf '%s\n' "$out"
}

# After two weeks, the ratio is your answer:
#   awk -F'\t' '{n[$2]++} END {for (k in n) print k, n[k]}' "$log"

If the hit ratio is low, you do not have a product yet. You have a habit that has not paid off. That is worth knowing before you write the landing page.

What changes for you

Before: you open a session, explain your setup, get help, close the session, and the explanation dies with it. Tomorrow the same explanation, in the same words, because you wrote them once and nobody kept them.

After: the explanation is written down once by the assistant itself, and read back before the next task starts. You notice it not as a feature but as an absence — the absence of that first ten minutes.

Our version of this is cachly: the assistant saves what it learned after a fix and recalls it before the next task, over MCP, so the memory survives restarts, model upgrades and switching editors.

An internal tool becomes a product the day you can prove it earns its keep for somebody who did not build it. Until then it is a habit with a README.

I build cachly — memory for AI coding assistants, over MCP.

ChatGPT and Claude remember your conversations. cachly remembers your system: the bug you fixed, why you chose Postgres, the deploy step that always breaks — and which decision it contradicts. Every assistant you use reads the same memory, and every lesson carries the name of whoever learned it, so nobody from your team has to learn it twice.

Free tier, hosted in the EU: cachly.dev

Four green CI checks. All four blind.

Heinrich Neb — Tue, 18 Aug 2026 13:44:42 +0000

Quick take

A broken CI check is loud: the pipeline turns red, someone looks. A blind check is silent — and silence renders identically to success.

We found four in one repo, in one day. All green:

A GitHub Actions lint job with timeout-minutes: 15 and --timeout=15m inside. The tool's timeout can never fire — the job dies first, and GitHub writes cancelled with no reason. It blocked our deploys for a full day.
A line-endings fix that never reached existing machines. .gitattributes applies at checkout: 1230 of 1455 files still had CRLF while git status said clean.
A guard that flagged its own comment, because it scanned prose as configuration.
A signup report that counted our own laptop as a "high purchase intent" lead.

Nothing rotted in the code. The ground moved underneath it: a runner, an old worktree, a comment, a generated domain.

The question that found all four: when did this check last say no? "Nothing is wrong" and "I cannot see anything" render identically on every dashboard — and only one of them is true.

The long version, with all four post-mortems: https://dev.to/heinrichneb/one-repo-one-day-4-ci-guards-that-were-green-and-blind-oge

I build cachly — persistent memory for AI coding assistants, over MCP. Your assistant re-reads your codebase every morning. It does not have to. Free tier, EU-hosted: cachly.dev

One repo, one day: 4 CI guards that were green and blind

Heinrich Neb — Tue, 18 Aug 2026 10:26:28 +0000

Guards that stopped guarding

Your test suite proves your code works. Nothing in your repository proves your checks still work.

A green check and a blind check look identical

We had a bad day. Not a dramatic one — nothing went down, no customer noticed. Just a day where four separate checks in the same repository were green, and all four had quietly stopped being able to fail.

That combination is worse than a broken check. A broken check is loud: the pipeline turns red, someone looks. A blind check is silent, and silence is indistinguishable from success. You keep merging, the dashboard stays green, and the thing the check was built to catch walks straight past it.

Here are all four, with the numbers, because the pattern only becomes obvious when you see them next to each other.

1. The timeout that could never fire

A lint job in GitHub Actions, configured like this:

jobs:
  lint:
    timeout-minutes: 15          # budget for the whole job
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-go@v6
      - run: golangci-lint run --timeout=15m   # budget for the tool

Both numbers are 15. Checkout, toolchain setup and module download all come out of the job's budget, so the tool's own timeout is unreachable — the job dies first.

That distinction is the whole point. A tool timeout fails with a sentence: timeout after 15m, package x. A job timeout kills the runner, and GitHub writes cancelled with no reason at all. Our deploy path was blocked for a full day across three commits, including one that only changed text, and not one of those runs said why.

The rule is one line: the job's limit must be larger than the limit of any tool inside it. Otherwise you have traded a diagnosable failure for a silent kill.

2. The fix that never reached the machine it was for

Months earlier we fixed line endings properly: * text=auto eol=lf in .gitattributes, verified, committed, done. It cost two days to find and it worked.

It came back. When we measured, it had never left: 1230 of 1455 text files in the working tree still had CRLF.

.gitattributes applies at checkout. A working tree created before the rule keeps its CRLF files forever, and nothing tells you — because git normalises when it compares:

git status         # "nothing to commit, working tree clean"
git diff           # empty
file config.ts     # ASCII text, with CRLF line terminators
prettier --check . # 287 files broken

In CI the same command reported two. Neither number is usable: believe the local one and you reformat 285 files nobody touched; ignore it and you miss the two that are real.

A fix that does not announce itself is not a fix for existing machines. We now have a check that says so out loud, and a one-liner that repairs it — no content change, no commit:

git rm --cached -r -q . && git reset --hard

3. The watcher that flagged its own comment

After fixing the timeout, we wrote a guard so it could not come back: read the workflow, and fail if any job's limit is not larger than the tool limits inside it.

It failed immediately. Not on the workflow — on the comment we had written above the fix, which quotes the old value --timeout=15m as part of explaining what went wrong.

The guard was scanning text and had no idea which lines were code. Two lines fixed it, and they are the interesting part:

for line in workflow.splitlines():
    if line.lstrip().startswith("#"):
        continue   # prose is not configuration

It is a small bug with an uncomfortable implication. A guard that reads text will eventually read the wrong text, and the failure mode is not "it misses things" — it is "it reports things that are not there", which trains everyone to ignore it. Then it misses things.

4. The report that counted us as customers

A bot message: 4 new company signups — high purchase intent. Four business email domains, worth reaching out to today.

One of them was provably our own laptop: our editor extension had started at 10:47:10 UTC and the account appeared at 10:47. The other three could not be attributed at all.

Two causes, both structural. Anonymous trial accounts get a generated address at <uuid>@trial.example.dev — our own domain — and the code that decides "is this a company?" only knew a list of free mail providers. Anything not on that list counted as a business.

And the signup handler recorded no origin at all. The clients had been sending an honest User-Agent for weeks. The server read it for abuse detection and threw it away.

So the report was not wrong about what it counted. It was counting something that could never answer the question it was asked.

The question that finds all four

None of these were found by running the checks. All four were green. They were found by asking a different question:

When did this check last say no?

That question separates "nothing is wrong" from "I cannot see anything", and those are the two states every dashboard renders identically. For a new guard it is cheap to answer — break the thing on purpose, watch it turn red, put it back:

# 1. break what the guard protects
sed -i 's/EXPECTED/WRONG/' config.yaml

# 2. the guard MUST fail here
npm run guard && echo "this guard is decoration" && exit 1

# 3. restore
git checkout config.yaml

Thirty seconds, once. If step 2 prints that line, you did not ship a guard — you shipped a green light with nothing behind it.

For guards that already exist, the answer is harder and more interesting: most repositories have no idea. There is no field anywhere recording when a check last rejected something. A guard that has been green for eight months is either protecting a very stable codebase or has quietly gone blind, and nothing in your tooling distinguishes those two.

We have not solved that part. The shape of a fix is obvious — one timestamp per guard, written every time it rejects something — but we have not built it, and I would rather say so than describe it as if we had. What we did do is start asking the question by hand, which is how all four of these turned up in a single day.

What this is really about

Every one of these four checks was written by someone competent, for a real risk, and worked on the day it was merged. Nothing rotted in the code. What changed was the ground underneath: a runner moved, a machine was older than a rule, a comment was added, a domain was generated.

Tests answer "does the code do the right thing?". Nothing in a normal repository answers "does the check still do its thing?" — and that second question has no owner, no runner, and no red light.

If you look for one thing after reading this: find the check in your pipeline that has been green the longest, and try to make it fail. You will learn something either way, and it takes about a minute.

Free tier, hosted in the EU: cachly.dev

My best-looking GitHub Actions run shipped zero installs

Heinrich Neb — Mon, 17 Aug 2026 20:28:16 +0000

Quick take

Your publish job is green. That proves you sent the file. It does not prove anyone can install it.

Every publish step I have ever written ended at the upload. The API accepted the request, the exit code was 0, the workflow went green, and I went to lunch.

A marketplace can accept an upload and then reject it in review, hold it in a queue, or list it under a version nobody sees. All of that happens after your job has already reported success.

Ours did exactly that for three weeks. We found out because a user asked why the version was so old.

The fix is one request at the end of the same job — ask the public API what the world can actually see, and fail if it disagrees with you:

PUBLISHED=$(curl -sf "$REGISTRY_API/my-package" | jq -r .version)
[ "$PUBLISHED" = "$VERSION" ] || {
  echo "uploaded $VERSION, world still sees $PUBLISHED"
  exit 1
}

Note the direction. You are not asking your own pipeline whether it succeeded — it already told you, and it was wrong. You are asking a stranger.

The long version — three weeks, two marketplaces, and the wording that made the job lie — is here: Your GitHub Actions run is green. Nobody can install your VS Code extension.

I build cachly — persistent memory for AI coding assistants, over MCP. Your assistant re-reads your codebase every morning. It does not have to.

Free tier, hosted in the EU: cachly.dev

Your CI is not flaky. It fails every 7 days.

Heinrich Neb — Mon, 17 Aug 2026 20:28:15 +0000

Quick take

Before you label a failing test flaky, write down the dates it failed. Flaky has no rhythm. Yours might.

"Flaky" is the most expensive word in CI. It closes the investigation. Nobody looks at a flaky test again until it fails on the day of a release.

Here is a cheap way to earn the label instead of assuming it. Pull the dates of the last failures and look at the gaps:

gh run list --workflow=ci.yml --status=failure \
  --limit 50 --json createdAt \
  --jq '.[].createdAt' | sort

Random failures scatter. If the gaps land on the same number over and over, something on a timer is doing this to you, and it is not the test.

Seven days is the number worth knowing by heart: GitHub Actions evicts a cache entry that has not been touched for a week. A dependency that is only ever restored, never rebuilt, disappears on a schedule — and the first run after that is slow, cold, and sometimes fails on a timeout that was always marginal.

That failure is not random. It has a calendar. Yours might be a certificate, a token, a log rotation or a nightly database refresh. Same shape, different clock.

The long version — how we found ours, and why the fix was not a longer timeout — is here: Your CI is not flaky. Your cache expires every seven days.

I build cachly — persistent memory for AI coding assistants, over MCP. Your assistant re-reads your codebase every morning. It does not have to.

Free tier, hosted in the EU: cachly.dev

Has your GitHub Actions gate ever said no?

Heinrich Neb — Mon, 17 Aug 2026 20:28:14 +0000

Quick take

Quick question. That guard you added to CI last month — have you ever watched it refuse anything?

Most of us test in one direction. We prove the thing works. We almost never prove the thing can fail.

So a guard gets written, the pipeline goes green, and everyone moves on. Green means the guard passed. It does not mean the guard would have caught anything. Those are different sentences, and only one of them is on the screen.

I ran this check on my own repo. Four guards. One of them had been green since the day it was merged, for a reason nobody enjoys hearing: it compared the wrong two values, so it agreed with everything.

The fix is not clever. Before you merge a guard, break something on purpose and watch it turn red. Thirty seconds, once, forever:

# 1. break the thing the guard protects
sed -i 's/EXPECTED/WRONG/' config.yaml

# 2. the guard MUST fail here
npm run guard && echo "the guard is decoration" && exit 1

# 3. put it back
git checkout config.yaml

If step 2 prints that line, you did not ship a guard. You shipped a decoration with a green light attached.

The long version — four guards, four different ways to be green and useless, and the one question missing from most acceptance criteria — is here: My own CI gate rejected me 4 times.

I build cachly — persistent memory for AI coding assistants, over MCP. Your assistant re-reads your codebase every morning. It does not have to.

Free tier, hosted in the EU: cachly.dev

My own CI gate rejected me 4 times: every one was the same mistake

Heinrich Neb — Mon, 17 Aug 2026 10:50:16 +0000

Proof over claim

You run the tests, they pass, you ship. Now try the other direction: build a check that must say no, and prove you have seen it say no. Most of us have never watched our guards refuse anything.

Four rejections, one shape

In one working day my own tooling stopped me four times. I kept the list, because by the third one the pattern was too clean to ignore.

One: I scanned a network range to find a service, found something answering, and concluded the service lived there. It was a leftover process serving an older model. The thing I needed was on a machine my scan never reached.

Two: I read an error message from a stored record and changed code based on it. The message came from a version that had been repaired three days earlier. I nearly rebuilt a fix that already existed.

Three: my own verification loop reported two passing test files as failing, because it searched output text for the word red and the German word for nevertheless contains those three letters.

Four: I added a file to a build without adding it to the file list. The build reported success, and the packaged result crashed on the first command.

The shape: an output is not a proof

Every one of the four was the same mistake wearing different clothes. Something produced output, and I treated the output as evidence about a different question.

A service answering proves something is listening. It does not prove what. A stored error message proves what a tool once said. It does not prove what the tool does now. A build that lists files proves the list is satisfied. It does not prove the result runs.

This is uncomfortable because output is exactly what we have. We read logs, exit codes and dashboards all day, and each of them answers a narrower question than the one in our head.

The gap between those two questions is where a whole day of work quietly goes wrong.

Why the checks caught it and I did not

Not because they were clever. Because each one asserted the narrow question in writing, and I could not talk them out of it.

The file-list check now starts the packaged tool and fails if it does not respond. It stopped caring whether the list was satisfied, which was never the interesting question.

The service check now asserts which model answers, not that something answers. One extra string comparison, and the leftover process becomes visible instead of convincing.

And every verification now reads exit codes instead of searching text. A word inside another word cannot fake an exit code.

The test most guards never get

Here is the part that took me longest to accept. For every guard I have written, I can tell you that it passes. For most of them, I cannot tell you that I have ever watched them fail.

A guard that has only ever been green is not a guard. It is decoration that happens to be the right colour, and it will keep being the right colour after it stops working.

So the acceptance for a guard needs two cases, and the order matters: first prove it says no when it should, then prove it says yes when it should. Written in that order, because the yes case is the one you will remember to write anyway.

When I applied this to my own package, the very first thing it caught was that the loop brake and the frozen-acceptance check both fired correctly. My tooling was fine. My test was wrong. That is the good outcome.

Add the refusal case to one guard today

Pick the guard you trust most. That is the right one, because trust is the thing being tested.

Step one: write a case that should be rejected, and assert the rejection. Not just a non-zero exit — assert the message names the reason, so a future failure for a different reason cannot pass as this one.

Step two: break the guard on purpose in a scratch copy and confirm your new case turns red. If it stays green, your case proves nothing and you have just learned that for free.

Step three: keep the broken-on-purpose command in a comment next to the case. Six months from now, that is the only way anyone can re-verify the test still bites.

The pattern in fifteen lines:

#!/usr/bin/env bash
# Guard acceptance: prove NO first, then YES.
set -u

# 1. The refusal case — the one almost nobody writes.
if out=$(./my-guard bad-input 2>&1); then
  echo "FAIL: guard accepted bad input"; exit 1
fi
grep -q 'reason:' <<<"$out" || { echo "FAIL: refusal without a stated reason"; exit 1; }

# 2. The happy case.
./my-guard good-input >/dev/null || { echo "FAIL: guard rejected good input"; exit 1; }

echo "OK: guard says no AND yes"
# Counter-check (run by hand once, keep the line):
#   sed -i 's/exit 1/exit 0/' my-guard && ./this-test   # must FAIL

That last commented line is the whole discipline. A test you have never seen fail is a rumour.

What changes for you

Before: your suite is green, and you cannot say which of those checks would notice if the thing they watch quietly stopped working. Green means nothing broke loudly.

After: for each guard you can point at the case that proves it refuses, and at the command that proves the case bites. Green starts meaning something specific.

In our own setup this is a hard rule: every finding becomes a check, and every check ships with the run that shows it failing. The record of those runs is what we keep in cachly, so the next session inherits the reason and not just the rule. Free tier, hosted in the EU.

Being rejected by your own tooling four times in a day is not a bad day. It is the day the tooling finally earned its keep.

Free tier, hosted in the EU: cachly.dev