DEV Community: Akash Goenka

What shipping coldstart to real users taught me

Akash Goenka — Tue, 04 Aug 2026 07:42:00 +0000

The hardest bugs weren't crashes. They were the ones where the system looked like it was working while quietly becoming less useful.

I've been shipping coldstart in the open for the last few months. Before that, I mostly used it myself, on my own repositories, with my own workflow.

I thought I knew where the weak spots were.

I didn't.

The first people who installed it found bugs I hadn't imagined, but more interestingly, they exposed assumptions I didn't realize I'd built into the system. Looking back through the issues and release history, almost every lasting fix came from removing one of those assumptions entirely.

Automatic capture wasn't enough

One of the first feature requests wasn't for a new capability.

It was for a manual one.

Coldstart captures notes automatically as you work, but users wanted a /capture-notes command they could run whenever they felt a session had reached a natural stopping point.

Adding the command turned up a much bigger problem.

Automatic capture marked files as "already offered." The manual command treated that flag as "already captured," so if you'd edited a file earlier in the session, the explicit capture command often skipped the most important work entirely. Brand new files could disappear from the worklist, while older read-only files stayed.

That forced me to rethink what those flags actually meant.

A file being offered to the agent once isn't the same thing as a good notebook entry existing.

The capture pipeline now tracks actual work instead of whether something happened to appear in an earlier prompt, and manual capture always prioritises the files you changed during the current session.

The feature request turned into a redesign of the capture model.

Measuring recall instead of tuning it

Another complaint was harder to pin down.

Recall is the part of coldstart that automatically surfaces older notes when they seem relevant to what you're working on. It felt noisy, but "felt" isn't something you can optimise.

So I stopped guessing.

I hand-labelled 140 real recall events and measured how often the injected note was actually useful.

The baseline precision was 31%.

Three different problems showed up.

Ordinary words like "status" or "merge" matched unrelated notes because they happened to share common vocabulary.

The same note could be injected repeatedly during one session, even though it was already sitting in the model's context.

And some of the query wasn't coming from the user at all. Harness telemetry and editor boilerplate were leaking into the relevance calculation and promoting notes that had nothing to do with the task.

The fixes were surprisingly small.

Require stronger evidence before accepting a single ordinary-word match.

Don't show the same note twice in one session.

Strip infrastructure text before building the search query.

Precision increased to 47% while keeping most of the genuinely useful recalls.

The biggest lesson wasn't the number.

It was that retrieval systems are very easy to make feel better and much harder to make measurably better.

Notes existed, but couldn't be found

One of the most surprising bugs wasn't that notes were missing.

They were there.

They just weren't retrievable.

The information that makes a note searchable—aliases, symbols, verified anchors—had drifted across multiple places. The capture prompt expected one shape. The documentation described another. The MCP tool had a third.

Everything still worked.

Until it didn't.

The solution wasn't another validation rule.

I froze the note schema in one canonical definition and generated every surface from it instead. The capture prompt, documentation, repair tooling and MCP interface all derive from the same source now, and tests fail if they drift apart.

That also led to kb repair.

Instead of assuming every notebook stays healthy forever, coldstart can now inspect itself and tell you which notes have become incomplete or difficult to retrieve.

That was a change in philosophy as much as implementation.

The notebook stopped being something users had to trust blindly and became something that could explain when it needed maintenance.

Context is temporary. State shouldn't be.

The most difficult bugs all had one thing in common.

They assumed an agent would remember something later.

Notebook capture originally handed the agent a worklist and a write contract in a single prompt.

Hours later, after conversation compaction, that prompt was gone.

The agent had to reconstruct the note format from memory.

Writing multiple notes had a similar problem. A malformed note halfway through the sequence stopped every note after it from being written, and there was no single place that could answer a simple question:

"Did this session actually get captured?"

The eventual solution wasn't another prompt.

It was moving the state somewhere durable.

The worklist now lives inside the repository itself, where the agent can read it again whenever it needs to. Writing happens in one batch instead of a chain of independent commands, coverage is calculated once, malformed notes are reported together instead of silently preventing everything after them, and the worklist updates itself as notes are completed.

The implementation is more complicated.

The behaviour is much simpler.

The fixes that lasted all looked the same

The installation problems, recall changes, capture redesign, repair tools and durable worklists all looked unrelated while I was fixing them.

Looking back, they all came from the same mistake.

I kept assuming information would stay synchronised.

That prompts would stay visible.

That different copies of a schema would evolve together.

That something shown once had effectively been remembered.

That writing a note automatically meant it would still be discoverable months later.

Real users broke every one of those assumptions.

Most of the fixes that lasted weren't bigger patches.

They removed the assumption altogether.

Replacing native dependencies with WASM removed an entire class of installation failures.

Generating every interface from one note schema removed documentation drift.

Persisting the capture worklist removed the need for an agent to remember instructions from fifty turns earlier.

Adding repair commands acknowledged that notebooks, like code, slowly drift and occasionally need maintenance.

That's probably the biggest thing shipping coldstart has taught me.

Bugs rarely come from the thing you were trying to build.

They come from the assumptions you forgot you were making while you built it.

coldstart is open source under the MIT license. If you end up trying it and manage to break an assumption I haven't found yet, there's a good chance it'll become the next entry in the release notes.

Codebase memory that can't lie: why your agent relearns your repo every session

Akash Goenka — Mon, 27 Apr 2026 04:33:40 +0000

Agents don't take advice. They take evidence. Here's what I deleted before I understood that.

I work with coding agents every day. And every day, in every new session, the agent starts over.

It greps around my repo. It opens files it doesn't need. It works out how the pieces connect — the same pieces it worked out yesterday, and the day before. Somewhere in there it rediscovers the one gotcha in the config loader that it had already traced, carefully, last week.

The agent isn't the memory. I am. Every session, I pour the context back in from my own head before we can do any real work. This is the codebase memory problem: without a persistent, verifiable record of what an agent already learned, every session pays the same tax again

The obvious fix doesn't hold

Write it down. Put it in CLAUDE.md, or AGENTS.md, or whatever your tool reads at startup.

I went further than that once. A whole docs/ folder — the important things about the codebase written out properly, so that any agent, or any human, could get oriented without me. It was good for about a week.

The problem was never writing it. The problem is that keeping it true is a job, and it's a job with no deadline, no ticket, and nobody chasing you for it. So it doesn't get done. The code moves, the docs don't, and now you have something strictly worse than no documentation: a confident, out-of-date map that an agent will follow straight into a wall.

I didn't want to write more documentation. I wanted context that maintains itself.

That's what I've been building. It's called coldstart, and this post is mostly about the things I built and then deleted along the way, because that's the part I'd actually want to read.

What version one taught me

The first version was an MCP server with four tools. It worked, sort of, and using it taught me three things I didn't expect.

Give an agent a free hand with a tool and it will abuse it. I assumed a capable model handed a good tool would use it well. It doesn't. Search has no natural structure, so the agent fires off grep after grep, each one an independent guess, trying to triangulate onto an answer from a scatter of weak signals. It's not stupid — it's doing exactly what you'd do with no map. But it's expensive, and it's slow, and it happens every single session.

So four tools became two. Fewer doors, less improvisation.

Agents don't believe bare tool output. This one changed the design. Hand an agent a ranked list of files and it will not act on it. It goes and checks. What makes it trust the result is seeing actual code alongside the ranking — the lines that matched, the symbol that's defined there, the receipts. Once the evidence is in front of it, it moves. Without it, the tool call was just a suggestion, and the agent quietly did the work itself anyway.

So find doesn't return a list. It returns a list with the evidence attached.

And agents guess line numbers. Watching sessions, I kept seeing the model reach for a file, decide roughly where the thing it wanted lived, and read the wrong range. Then read again. So the second tool, gs, exists to give the shape of a file cheaply — the symbols, their line ranges, who calls them — so that when the agent finally does a real read, it reads the right twenty lines instead of guessing at two hundred.

Both of those are the same lesson wearing different clothes, and it's the subtitle of this post. The agent doesn't want to be told. It wants to be shown.

The graveyard

Coldstart got simpler as it got better. Almost every clever idea I had died, and the interesting thing is that most of them died the same way.

Making the ranking clever. The first ranker had more signals in it than the one that shipped. The one I was most attached to was graph centrality — the intuition being that a file lots of other files import must be an important file, so it should rank higher. It sounds obviously right. It's wrong. The most-imported files in any repo are the utils, the constants, the type definitions. Centrality quietly promotes the least useful results in the codebase. I started calling it the hub trap, and then I deleted the signal. What's left is deliberately blunt: how many of your query terms does this file actually cover, and does it define them or merely mention them.

Nudging the agent's decisions. If it won't read my descriptions, maybe it'll respond to a nudge at the right moment. It won't. Advisory rules get ignored, or pattern-matched around, or they steer behaviour somewhere I didn't intend. Every attempt to persuade the agent failed, and the ones that "worked" only worked because they'd become hard constraints rather than advice. That's when surface, don't steer stopped being a philosophy I liked and became a finding I had to live with.

Freshness by timer. Both the index and the notes originally aged out on a TTL, the way a cache does. This is wrong, and it took me longer than it should have to see why: time doesn't make knowledge stale. Code changes do. A note written six months ago about a file nobody has touched since is perfectly good. A note written this morning about a file you refactored at lunch is garbage. TTLs went in the bin and got replaced with content hashing.

A "map my repo" command. The tempting onboarding feature — one command, point an LLM at the whole codebase, generate notes for everything. I rejected this one before building it, and I'm glad. Batch synthesis produces exactly the thing the notebook exists to prevent: confident prose that nobody verified, generated by a model that was never actually solving a problem in that file. It would have mass-produced the failure mode, with a bill attached.

The pattern in all four graves is the same. Every time I tried to make coldstart smarter, the answer was to make it more honest instead.

The detour I didn't take

There was a point where I nearly went in a completely different direction.

I'd noticed agents re-reading the same files across sessions, and I thought: the transcripts are sitting right there on disk. Scrape them. Work out which files turned out to matter for which kinds of question, and build an index of that. It's local, it's deterministic, it fits everything else I believe about this problem.

I didn't build it, and the reason I didn't is the reason the rest of this project exists.

Capturing what an agent learned is the easy half. Keeping it true is the hard half. A big pile of harvested knowledge with no mechanism to invalidate it isn't memory — it's a faster way to be confidently wrong. And the code underneath it moves every day.

So I went at the hard half first.

What survived: codebase memory, not a journal

The notebook. It's the part I care about, and it rests on one distinction:

It doesn't store a journal. It stores what an agent learned about the codebase, anchored to the codebase.

That's not a small difference. Most "give your AI memory" tools are storing a record of your conversations — what was said, what was done, in what order. That decays into archaeology. What I want isn't a diary of past sessions; it's the durable facts about this repo that a session happened to uncover. How a flow moves across files. Why a piece of code is the way it is. The trap that isn't visible from reading the code, only from having been burned by it.

Here's how it holds together.

An agent writes notes at the end of a task — while it still has the full context, having actually read the code and fixed the thing. Not later, not in a batch, not by a model that was never there.

Every note carries anchors: the real files and symbols its claims rest on, stamped with a content hash at the moment it was written. The index re-checks those stamps as the code changes. When the evidence moves, the note says so — it renders as [evidence changed], drops down the search results, and degrades into a labelled hypothesis instead of continuing to assert a fact it can no longer support.

That's the whole trick, and it's why the notes can be trusted at all. The codebase is the source of truth, and a note that no longer matches it admits it.

Writes go through a gate, so a new note is checked against what's already there and either merges into an existing one or is explicitly declared new. Notes don't pile up into a swamp.

You can just look at it

None of this is a black box you have to take on faith. coldstart kb view opens the notebook in your browser — every note the agents have written, what each one is anchored to, and whether it still holds.

Here it is on Arches, an open-source (AGPL-3.0) heritage data platform I pointed coldstart at as a test. A large Django codebase I have never worked on.

115 notes and 22 flows, written by agents doing real tasks in a repo they'd never seen before. The note open here is a hub note on models.py — a file too big for one summary, so it's split into facets, one per symbol.

Look at what a facet actually says. Not "this is the ResourceInstance model" — I can get that from the class name. It says that get_instance_creator() quietly falls back to a settings default when no EditLog record exists. That's the kind of thing you only learn by having been in there, and exactly the kind of thing you've forgotten by next Tuesday.

Above it sits the anchor block: the verified code addresses those claims rest on. Every note in this notebook is fresh, because nothing has moved under them since they were written. Change one of those files and the note flips to [evidence changed] — here, in search results, everywhere it surfaces. The green dot is load-bearing, not decoration.

And the framing I keep coming back to, because it's the honest one:

Written by agents. Kept honest by the codebase.

Not "the agents maintain it, so you don't have to." That would be a lie, and everything in the section above about version one tells you why — agents given a free hand produce junk. The notes are trustworthy because there's a mechanism adjudicating them. Take the mechanism away and you've just automated the production of rotting markdown.

And it accrues. This is the part that only shows up over time. One note is a convenience. A few hundred, written across months of real tasks and continuously re-checked against the code, is something else — a picture of the codebase that no single session could have produced, built by the agents that were actually in there doing the work. The second time a question comes up, the answer is already sitting there.

And it can be shared. The notebook is private to you by default — nothing leaves your machine unless you say so. But if you want it, one command commits it to the repo as an append-only log that merges cleanly across branches, so parallel work doesn't collide. Then it isn't your agent's memory any more, it's the team's. Your colleague's agent starts from what yours already worked out, and the notes keep getting checked against the same code you're all changing.

The boring layer underneath

Under the notebook sits the navigation, and it's deliberately dumb.

find answers "which files are about this?" and gs answers "what is this file, and who uses it?" Both run against a deterministic index built by static analysis — file paths, symbol names, exports, the import graph. Ranking is by how many of your query terms a file actually covers, weighted toward terms that name something declared in it. No embeddings. No model in the query path. No service to run. You can always see why a file came back.

"Static" here means static analysis, not stale: a background process watches the repo and updates the index within seconds of an edit, uncommitted ones included.

There is no API key anywhere in coldstart, and no bill. Nothing is sent off your machine — not to me, not to anyone. The index is parsing, not inference. And the notes are free too, in the sense that matters: the agent writes them as part of the task it was already doing, out of the context it already had in front of it. There is no second model reading your codebase to summarise it. That was the whole point of refusing batch synthesis — the moment you need a separate pass over the repo to generate knowledge, you have signed up for a cost, a delay, and a model forming opinions about code it was never asked to fix.

It's in this post for one reason only. It's the thing that makes the notes verifiable. The notebook's freshness stamps come from the index, and without them the notes would just be a wiki with better branding.

Where this leaves me

I wrote a lot to get here, and most of it wasn't worth keeping. Four tools became two. Descriptions, nudges, TTLs, batch synthesis — all built or seriously considered, all gone.

Which is, I've come to think, the same argument the notebook is making. What's worth keeping was never the transcript of how you got somewhere. It's the residue that survived contact with the code.

**coldstart is a codebase memory and navigation layer for coding agents. Open source, MIT, pure Node.

npm install -g @cstart/coldstart
cd your-project
coldstart init

Works with Claude Code, Cursor, and Codex.

Site: https://coldstartmcp.dev/
GitHub: https://github.com/AkashGoenka/coldstart
npm: https://www.npmjs.com/package/@cstart/coldstart

If you try it, I'd genuinely like to know where it falls over. Most of what's in this post came from me and a couple of friends using it and something breaking.